Scaling up a Django + React Web Scraping Website
$30-250 USD
Paid on delivery
I am working on a platform that runs a Django/Postgres backend with a React frontend. The platform scrapes data from multiple sources with Scrapy, cross-relates them, and gives some basic analysis of them via the front-end and HighCharts.
I am running into some issues with scaling the platform and some strange issues getting the entire system to work together. For example, running Scrapy & Twisted on a worker server has been problematic at times and I am having issues using Python multi-processing in tandem with the Django ORM (objects appear to be created randomly).
Project ID: #26870645
About the project
6 freelancers are bidding on average $161 for this job
Can help in scaling the scraper with scrapy. want to know more about it. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation w More
Why not move your script to AWS ec2 instead if you want it to run smoothly. If that doesnt work then the problem is from the script not the server
