Create scraping code or tool to collect article metadata from top US News websites, blogs and magazines. Scrape 10+ years of article metadata. Auto-scrape new article metadata weekly.

We're building a tool to analyze seasonal trends for web articles. We need a large bank of article metadata going back as far as possible (preferably 10+ years) for many news sources. We would like to start with at least 100 top news sources / magazines/ blogs , with the potential to expand to more sources down the line. Popular Websites that are primarily blogs and aren't attached to a newspaper are also great for this project (like cnews, techcrunch, medium, gawker, marginal revolution, etc). The information/metadata that we need for each article goes as followed: Article URL Article Title Article Date (highly preferred, but I understand) Article Sub-Title (if available) Article Description (if available) URL to article Header Image (if available) Project will be split into 3 phases Phase 1) Generate a list of 100+ news sources that you can easily scrape Phase 2) Scrape articles going back as many years as possible -- preferably 10+ years or since the inception of the website-- we want as much data as possible. Phase 3) Deliver tool that will regularly scrape the bank of news sites from phase 1 for new articles And then afterwards, we talk about a continued engagement where we build upon the bank of news sites from phase 1. The primary difficulty with this project is getting around bot / web scraping detectors and finding a way to get many articles. Using pagination and scraping home pages doesn't seem to work I feel like looking at sitemaps would be a good place to start for this. Theoretically web crawlers and spiders are going through every webpage of every news source, and that might be analogous to what is required to complete the project. However, that's just a suggestion. RSS feeds are another thing to look at. Let me know if you have any questions