Automated Web Scraping in PHP or Python saved in MySQL Database

Awarded Posted Jun 16, 2013 Paid on delivery
Awarded

I need code that gets information scraped from many websites and updates a MySQL database as well as storing images.

I will supply a MySQL DB Diagram with design and fields needed to be scraped. It can be modified if needed.

It will be hosted on Amazon EC2 so I will have flexibility in code/plugin support.

I need the most efficient form of scraping to the MySQL DB, saving images, and compressing the images for the web. I do not want a full GUI webkit running. Any library for scraping/image compression that is not processor intensive.

Images will be hosted on Amazon S3. If that's a quick addon to save images there after compressing, that would be great!

Websites scraped are simple html to javascript/aspx. This will be an ongoing thing as more websites will needed to be scraped.

I need someone I can work to guide me in DB design and making sure it's efficient. Also if there are better options than PHP/Python/Amazon, suggestions would be great! Shouldn't take more than 5 minutes of chit chat. I will supply one website to scrape and my DB design and work from there!

It will be ran daily or multiple times a day. It will rescrape the site and skip/update existing entries and add new ones.

MySQL PHP Software Architecture

Project ID: #4630632

About the project

13 proposals Remote project Active Jun 19, 2013

13 freelancers are bidding on average $14/hour for this job

kalpataru44

I can do this, Please allow me to work for you

$8 USD / hour
(53 Reviews)
7.1
mmadi

Iam ready to start

$12 USD / hour
(46 Reviews)
7.1
tzo

Can help you on this.

$15 USD / hour
(301 Reviews)
7.0
phpXpertbd

Seasoned web scraper. I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$12 USD / hour
(64 Reviews)
7.1
NayaPakistan

Hi, I have advanced web scrapping solutions. Please provide more details and lets start it!

$11 USD / hour
(288 Reviews)
6.8
ithinksolutions

Thanks for inviting us to bid on this project. We have a team of highly qualified and creative professionals. Give us a chance to show our talents and we assure you quality.

$12 USD / hour
(74 Reviews)
7.3
nitelfreelance

+13 years of experience in Python scripting.

$12 USD / hour
(31 Reviews)
6.0
softnology

Please check PMB

$14 USD / hour
(1 Review)
5.5
workbeezcom

hi, i can help you easily on this project let's discuss more.

$38 USD / hour
(14 Reviews)
5.2
ebin123456

i am ready to do this project

$8 USD / hour
(14 Reviews)
4.2
rjr121

Dear sir. i can do it. please check your PM.

$11 USD / hour
(5 Reviews)
4.3
webdevit

php/mysql team

$14 USD / hour
(6 Reviews)
2.6
azaam90

What type of information are you looking to scrap and how many websites I just would like an overview to see it in my head more clearly... Thanks

$12 USD / hour
(0 Reviews)
0.0