Crawl Domain Find Expred Domains
$10-30 USD
Paid on delivery
We are looking for a crawler to crawl every page of a website looking for external links pointing to expired domains.
User should definde a list of sites to crawl via text file. Crawler should work logically crawling all pages of a site and not be sitemap dependent. Only unique external domains should be logged to prevent duplicate domain availability lookups.
User should also be able to define a list of urls to ignore checking for availability; eg. [login to view URL] etc. these domains should be user defined in a blacklist text file.
Results should be given in a csv file listing linking domain and available domain.
Project ID: #19760507
About the project
4 freelancers are bidding on average $74 for this job
Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO More
Hey, I think i may be able to help you out on this project since i have worked on a similar project at work. Im new to freelancer, though i have years of experience in software development in python.