Find Jobs
Hire Freelancers

Puppeteer JS Site Crawler(How much experience do you have with Puppeteer?)

$250-750 USD

Closed
Posted about 3 years ago

$250-750 USD

Paid on delivery
I’m looking to build a crawler to perform a few meta and performance checks across multiple similar websites. This crawler should be built with Puppeteer and with clear unit tests. The intent is to crawl the homepage of a site and its site map (well defined), and then a long list of sub pages which are defined on that site map (and linked sub site maps). These pages are grouped into 3 well defined categories and are generally the same. We will then run a specific list of checks across each of these page types, and another set of checks across every page globally. The checks that I have defined and will share at the beginning of the project are quite simple, such as “does a <meta> description tag exist” and “does the page title contain some specific string”. The reason I am interested in Puppeteer (or Foxr) specifically is because I’d also like to measure things like page load time (Largest Content Paintful) and track heavy resources (such as images and scripts). To summarize, I need a web crawler that, with an original sitemap url, will crawl all sub-sitemaps and then the pages listed within those (strictly 3~ levels deep). Once on each of those pages, we will run a series of checks (some global, some category specific) and return details on the result (sometimes a boolean, sometimes and integer). I have fully documented this internally and will share upon beginning of the project. I’m also technical and will be reviewing your code and unit tests. To show that you’ve read this description entirely, please include the word “dinosaur” in your response. I will be happy to jump on a call with you during development to answer any questions you may have. Thank you!
Project ID: 29127417

About the project

6 proposals
Remote project
Active 3 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
6 freelancers are bidding on average $558 USD for this job
User Avatar
Hi there, I'm Matt Sergei (do call me Matt, please) and would like to help you - yet am currently busy probably till Thursday, Feb 11th. I've read your requirements and have experience with Puppeteer - just recently scripted a Google Cloud function to output the PNG image of a web page drawn Leaflet Map (and some other functions in the past). So do provide more details - I guess you want URLs to pass, checks and some other parameters to be read from a file (or a JSON request, can be in URL parameters) and return results back as ... what? Also JSON file etc. Meaning I plan to create a Node.js (function) code to crawl a website passed to it. Otherwise I'm not a dinosaur though am webmaster and developer for 26+ years. Have not done extensive unit tests in the past yet I believe you envision how they should be created and run. Regards and hope you're safe, Matt
$750 USD in 12 days
4.8 (41 reviews)
5.9
5.9
User Avatar
Hi there in Malaysia!!! I'm a software developer with many years of hands-on experience. I've completed many small and medium sized projects over those years. My recent accomplishments is a server that integrates 6 third-party platforms into a single interface. I used puppeteer to log to one site which uses the recaptcha technology to filter bots. I wrote fully documented and tested code. I used Yahoo!!'s yuidoc to generate source code documentation and jest for unit testing. The end result was modular and robust code. I will use those core principles to deliver your important project. I read your description: “dinosaur”. Looking forward to hearing from you. Regards.
$750 USD in 10 days
5.0 (11 reviews)
5.4
5.4
User Avatar
"dinosaur"Hi Muhammad Solihin. Thanks for your job posting. I just read your main idea carefully and it catches my eye. I found that I fit best for you because my skills and experiences are exactly fit to your requirement. I have rich experiences dealing with crawler building. So you can believe about the project and take your busy time . If you award me, you will just get a result. Among thousands of specialists in this sector, what differs us from others is our problem solving methods. I can contact the anytime you want and work full-time for you. Looking forward to hearing you back and discussing all details. Best Regards, Nikita.
$250 USD in 7 days
5.0 (2 reviews)
3.4
3.4
User Avatar
Dinosaur Hi I know puppeter and worked in .net so I can implement this. I am a technical architect by profession having 10+ of experience in below technologies: SharePoint Online / SPFX / Powerapps / Workflows ASP.NET C# .NET Core Angluar / React Js/ Typescript JavaScript / jQuery HTML / CSS Web Development / Web Scraping using .NET And Python Azure Python AWS Please let me know if we can discuss this over call or chat. Your expectations will be met for sure. Thanks, Hitesh
$700 USD in 8 days
5.0 (3 reviews)
2.7
2.7
User Avatar
I've built several projects with Puppeteer including a twitter bot, automated tests and I can confidently accept this project! btw I'm not a bot I'm a "dinosaur".
$650 USD in 10 days
5.0 (1 review)
1.7
1.7

About the client

Flag of MALAYSIA
BALIK PULAU, Malaysia
5.0
12
Payment method verified
Member since Sep 25, 2020

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.