Stagecoach Software Pty Ltd has a fully functioning ruby Nokogiri scraper populating a MySQL database from a particular website. The scraper is based on a cloud server with cPanel interface and runs on daily cron jobs. The scraper generates in the order of one thousand rows of data per day. The database is being built to function as a backend to a forthcoming website.
The company requires the scraper to be adapted to operate on a second website operating in the same industry and displaying the same type of data.
The existing code is available on github and Heroku and the development database is available on Heroku. These may be inspected by approved bidders before they accept the project.
Sundry other minor changes to the code and database may be required as the project progresses to help achieve the corporate objective of providing a useful and functioning website. The freelancer must ensure that the current production database and existing scraper continue to operate without interference throughout the project.
Further written instructions including identification of the site to be scraped will be provided to approved bidders. Bidders are free to revise their bids upon studying the instructions, the code or the existing database.
This project follows freelancer Project ID’s 5477803, 5129765, 5066285 and 4975478. Further similar scraping projects for other sites will be offered to the successful bidder if this project is completed successfully. Those scraping projects will be followed by the creation of a RoR website that will present graphs showing aggregate and individual data obtained from the database and allowing complex user searches. Preference will be given to freelancers interested in and qualified to carry on work in the further backend projects and subsequent building of the website.
Bidders without demonstrable experience in Ruby, MySQL and Web Scraping will be rejected. Bids placed within minutes of the project being posted will also be rejected as indicative of no thought being applied to the project requirements.
Please do provide details of the github and heroku locations to be able to check the existing system. Also please do let me know the exact changes required and the new target to be scrapped from. Looking forward to working on this with you
Hello,
I gone in your posting details and I can do this if you choose me for this task on budget and time.
I am having 8+years of Experience with Design/Development and you can see my 100% complete rate and good feedback and ratings from clients here on freelancer And also Expert in Module creation and customization any task related Php/.net(web/desktop)/Java(Mobiles apps development/games/flash) framework
I did 1000+ websites/applications and i am sure you will get best work in cheapest cost even i will give you unlimited updates until you get satisfied.
For more details on your task and to see my similar portfolio please contact me.
Waiting for your valuable response!
Thanks
Hi
What do you use for scraping? mechanize? httparty? or just plain nokogiri?
Is it possible to check code?
Are you sure that heroku can handle increased load?How many workers do you have now?
Why do you use mysql with heroku?
How do you test your code? Quality of scraping?
--
Ok. I did some scraping jobs before, with mechanize gem. Also tried httparty.
Can do this project on a weekend.