Puppeteer JS Site Crawler(How much experience do you have with Puppeteer?)
$250-750 USD
Closed
Posted about 3 years ago
$250-750 USD
Paid on delivery
I’m looking to build a crawler to perform a few meta and performance checks across multiple similar websites. This crawler should be built with Puppeteer and with clear unit tests.
The intent is to crawl the homepage of a site and its site map (well defined), and then a long list of sub pages which are defined on that site map (and linked sub site maps). These pages are grouped into 3 well defined categories and are generally the same. We will then run a specific list of checks across each of these page types, and another set of checks across every page globally.
The checks that I have defined and will share at the beginning of the project are quite simple, such as “does a <meta> description tag exist” and “does the page title contain some specific string”. The reason I am interested in Puppeteer (or Foxr) specifically is because I’d also like to measure things like page load time (Largest Content Paintful) and track heavy resources (such as images and scripts).
To summarize, I need a web crawler that, with an original sitemap url, will crawl all sub-sitemaps and then the pages listed within those (strictly 3~ levels deep). Once on each of those pages, we will run a series of checks (some global, some category specific) and return details on the result (sometimes a boolean, sometimes and integer).
I have fully documented this internally and will share upon beginning of the project. I’m also technical and will be reviewing your code and unit tests. To show that you’ve read this description entirely, please include the word “dinosaur” in your response. I will be happy to jump on a call with you during development to answer any questions you may have. Thank you!
Hi there, I'm Matt Sergei (do call me Matt, please) and would like to help you -
yet am currently busy probably till Thursday, Feb 11th. I've read your requirements and have experience with Puppeteer - just recently scripted a Google Cloud function to output the PNG image of a web page drawn Leaflet Map (and some other functions in the past).
So do provide more details - I guess you want URLs to pass, checks and some other parameters to be read from a file (or a JSON request, can be in URL parameters) and return results back as ... what? Also JSON file etc.
Meaning I plan to create a Node.js (function) code to crawl a website passed to it.
Otherwise I'm not a dinosaur though am webmaster and developer for 26+ years. Have not done extensive unit tests in the past yet I believe you envision how they should be created and run.
Regards and hope you're safe,
Matt
Hi there in Malaysia!!!
I'm a software developer with many years of hands-on experience. I've completed many small and medium sized projects over those years. My recent accomplishments is a server that integrates 6 third-party platforms into a single interface. I used puppeteer to log to one site which uses the recaptcha technology to filter bots. I wrote fully documented and tested code. I used Yahoo!!'s yuidoc to generate source code documentation and jest for unit testing. The end result was modular and robust code.
I will use those core principles to deliver your important project.
I read your description: “dinosaur”.
Looking forward to hearing from you.
Regards.
"dinosaur"Hi Muhammad Solihin.
Thanks for your job posting.
I just read your main idea carefully and it catches my eye.
I found that I fit best for you because my skills and experiences are exactly fit to your requirement.
I have rich experiences dealing with crawler building.
So you can believe about the project and take your busy time .
If you award me, you will just get a result.
Among thousands of specialists in this sector, what differs us from others is our problem solving methods.
I can contact the anytime you want and work full-time for you.
Looking forward to hearing you back and discussing all details.
Best Regards,
Nikita.
Dinosaur
Hi I know puppeter and worked in .net so I can implement this.
I am a technical architect by profession having 10+ of experience in below technologies:
SharePoint Online / SPFX / Powerapps / Workflows
ASP.NET C#
.NET Core
Angluar / React Js/ Typescript
JavaScript / jQuery
HTML / CSS
Web Development / Web Scraping using .NET And Python
Azure
Python
AWS
Please let me know if we can discuss this over call or chat. Your expectations will be met for sure.
Thanks,
Hitesh
I've built several projects with Puppeteer including a twitter bot, automated tests and I can confidently accept this project! btw I'm not a bot I'm a "dinosaur".