Check for Valid URLs Rapidly - Using Rotating IP and Proxies - Mac OS

$30-250 USD

Closed

Posted

over 7 years ago

$30-250 USD

Paid on delivery

I need a piece of software, preferably desktop based for Mac OS, that cycles through an increasing integer and enters it into a URL. The software should check if the URL leads to a valid product page, and if so, saving the integer in a list within the software's GUI. The general template would require the base site address, and the starting integer. Both numbers defined by the user. For instance: http://(sitenamehere)/cart/(integer here):1 So (sitenamehere) would be the site we're scraping info on, and (integer here) would be the number we are starting at. The software should increase by one integer for every check, and should use Rotating IPs and proxies to avoid bans. I would like to software to perform these checks as quickly as possible, looking to check about 800k-1 million product pages per day.

PHP

Software Architecture

Web Scraping

Project ID: 11128270

About the project

7 proposals

Remote project

Active 8 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

7 freelancers are bidding on average $152 USD for this job

@e3d

Hi, it's not a problem to make such software, however sites usually ban your ip after few hundred-thousand attempts. this can be avoided by using proxies, but it's not cheap to hold your own proxies, so I can quote you per-million pricing if you'll tell me which website you're scraping

$147 USD in 3 days

5.0

(344 reviews)

9.0

@phpXpertbd

Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy Proshanta

$250 USD in 5 days

4.9

(101 reviews)

7.5

@hAbd0u

Hello, You project were very clear, but I have some questions: 1) "The general template would require the base site address, and the starting integer. Both numbers defined by the user. For instance: http://(sitenamehere)/cart/(integer here):1", is this going to be just one site to test? 2) "I would like to software to perform these checks as quickly as possible, looking to check about 800k-1 million product pages per day.", this will generally need a very high speed connection, like the VPS or a dedicated server has, so are you open for that?

$190 USD in 10 days

5.0

(7 reviews)

4.5

@developers786

because I have developed the scraper before. for yellowpages, justdial and dx.com. auto increment in pages to scrape next page data and proxy script is implemented. I can provide you the software after minor changes. because your said requirement is already implemented.

$133 USD in 10 days