Find Jobs
Hire Freelancers

Build 2 further web scraping tools for existing MySQL database - repost

$250-750 AUD

In Progress
Posted over 10 years ago

$250-750 AUD

Paid on delivery
Stagecoach Software Pty Ltd requires two automated ruby web scraping tools to populate an already existing MySQL database. This is an extension of the recently completed project No. 4975478 which has already developed two scraping tools. A third and fourth tool are now required. The third scraping tool needs to run on an automated schedule every day. It is required to scrape a single large web site listing thousands of unique products and then populate the MySQL database with details of the products and prices sought. The details to be scraped appear in a mostly consistent format on the site. On the first scrape the tool must obtain details for all products listed whilst subsequent scrapes are required to track new products listed or changes of price to existing products. The freelancer must populate the DB with the scraper as part of the project. It is essential that the new tool does not duplicate data already being obtained from the existing scraping tools. The fourth tool is required to automatically download and scrape on a weekly basis a different website that lists the realized prices for a significant proportion of the products being captured by the first and third tools. The fourth tool must populate an existing table in the MySQL database with these products and price results. Again, duplication of data already obtained must be avoided. The MySQL database already combines data from the first two tools to calculate the difference between realized prices and initial listed prices. The freelancer must ensure that the database continues to calculate these differences using the freshly scraped data obtained with tools 3 and 4. The DB is hosted on a cloud platform. The freelancer must ensure that the first and second tools continue to operate with the existing db while the third and fourth tools are being developed and tested. The new scraper tools must be able to be easily activated from the host site and full written operating instructions must be provided. Full details of the site to be scraped and the required DB columns will be supplied to the successful bidder. This project is to be followed with the creation of a RoR website that will present graphs showing aggregate and individual data obtained from the database and allow user searches. Preference will be given to freelancers interested in and qualified to carry on into this third stage.
Project ID: 5066285

About the project

10 proposals
Remote project
Active 10 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Hey David, The quote for your items is: 1. and 2. - $450 3. parsing results by scraping additional site - $150 4, 5, 6 - let's say an additional $100 Total: $700 ( + freelancer.com fees) AUD
$777 AUD in 14 days
5.0 (7 reviews)
3.1
3.1
10 freelancers are bidding on average $611 AUD for this job
User Avatar
Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi
$250 AUD in 5 days
5.0 (284 reviews)
7.5
7.5
User Avatar
I'm a full time consultant, based out of my home office in Houston, Texas, United States. Even though I'm in the United States, I generally am fully available for communication during your business hours (I'm commonly online through Australian 6:00PM EST) I've been a member of Freelancer.com since 2001. In the past several years, scraping tools have been my bread and butter. I've written scrapers in C#, ColdFusion, JavaScript/node.js, and Ruby/Rails. I've leverage several different libraries: Selenium, straight HTTP, iMacros, Mechanize, and more. The primary project I've worked on involved scraping tens of millions of package shipments, estimates, and tracking, all while keeping up with the changes made to those websites. These scraping efforts involved not only grabbing data, but also downloading files (very difficult, and not something that most typical scraping technologies allow for). As for your project, as you've described it, it sounds like we can use Mechanize. If necessary, we could use Selenium in a cluster. Reducing duplication shouldn't be an issue; I assume the scraper can run on Rails as well? I'm making some assumptions about your hosting setup; perhaps we could discuss further.
$750 AUD in 3 days
4.7 (3 reviews)
2.5
2.5
User Avatar
I have fifteen years experience developing software using Linux, PHP, CakePHP, Ruby on Rails, MySQL, Bash, SSH, Git and emacs. I like install, configure and update my own servers, VPS or dedicated, preferably Debian or OpenBSD.
$666 AUD in 4 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of AUSTRALIA
Idalia, Australia
5.0
57
Payment method verified
Member since Feb 25, 2013

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.