Data mining - download 18 million webpages sourcecode

Closed Posted Aug 21, 2013 Paid on delivery

$30-250 USD

Paid on delivery

Closed Paid on delivery

I need to download 18 million webpages from one website. Provider can save the source code in html or txt.

One file per webpage.

The hard part of this project is the volume. I tried downloading using internet download manager and other downloading tool and the website will become 403 if it senses over usage and using proxy is slow.

I can provide the 18 million web urls in txt, sql, or whatever of you choosing. Once completed, you may compress the files and upload it to an FTP account which I will provide to you.

Time of delivery is the key so provider must have previous experience in past projects alike to show that you are capable of delivering.

Data Mining Web Scraping

Project ID: #4850011

About the project

9 proposals Remote project Active Oct 14, 2013

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

9 freelancers are bidding on average $461 for this job

dablu11's Profile Picture
dablu11

=== Please check PM for details ===

$550 USD in 40 days
(205 Reviews)
6.8
flashsaiful's Profile Picture
flashsaiful

Hi, ready to start

$2222 USD in 15 days
(143 Reviews)
6.7
abupabuya's Profile Picture
abupabuya

hi sir.i have a list , pls check my sample at message inbox

$200 USD in 1 day
(83 Reviews)
6.1
ashok7925's Profile Picture
ashok7925

Hi, Please check your inbox.

$150 USD in 25 days
(25 Reviews)
5.1
pilotarif's Profile Picture
pilotarif

Hope I am the right person to whom you are looking for your project. I believe actions are more effective than words. I value my clients' opinions, ideas and their passion to meet their goals. I look forward to helping More

$277 USD in 15 days
(12 Reviews)
4.1
PawelFlorek's Profile Picture
PawelFlorek

I can do this.

$231 USD in 18 days
(1 Review)
0.9
matthewsandrews1's Profile Picture
matthewsandrews1

I can do this please consider my bid, I can do it without a time out or being blocked

$94 USD in 15 days
(1 Review)
2.4
Flopet17's Profile Picture
Flopet17

Hi, I need more information please!

$150 USD in 30 days
(0 Reviews)
0.0
nmlemus's Profile Picture
nmlemus

I can try using grid computing to download all the pages ASAP, send me a file with the urls. Regards Noel

$277 USD in 3 days
(0 Reviews)
2.5
Post a project like this