Create a Fast Web Crawler to Extract Google SERP Data


I need someone to develop me a fast and powerful web crawler to extract data from Google SERP.

Important key features;

1. Must have a user interface protected by user and password where I can manage my campaigns.

2. Must have rotating proxies support

3. Multi threading support

I want a dashboard, where I can create/edit/delete my campaigns. For every campaign, I will set a google query (ie: intitle:"fashion" "blog") and decide how many results I want to extract.

When I will click on the "Start Crawler" button, I want to see the results in real time.. so If I decided to crawl the first 100 000 results for this query, if the crawler have extracted 1000 results, it should show 1000 results extracted out of 100 000.

Once the extraction is completed, I want to receive an email that notify me that the campaign has finished.

Once the process is completed, I want to have 3 options;

1. Remove duplicated domains (Remove all entries that is for the same domain) ie:

[login to view URL]

[login to view URL]

[login to view URL]

Should keep just first entry so "[login to view URL]" and delete all the other entries.

2. Update metrics (This will use the SEMrush API to update the keywords / organic traffic for every extracted domains)

3. Export the data in Excel format. (Columns; Full URL / SEMrush Keywords / SEMrush Organic Traffic)

Skills: Web Scraping, PHP, Python, Javascript, Software Architecture

See more: extract google analytics data limit, php extract google serp, extract google analytics data, web crawler extract description php mysql, extract google analytics data xml, fast web crawler mysql, web crawler extract data, java web crawler extract web page title, extract google analytics data daily, web crawler extract data website, web crawler extract phone number, web crawler extract data excel, extract google play data, web crawler script google, code web crawler extract data website, web crawler extract information java, program web crawler extract information website, create simple web-crawler and email sender, how to create a web crawler

About the Employer:
( 35 reviews ) Sabrevois, Canada

Project ID: #23957254

18 freelancers are bidding on average $594 for this job


Hey there, I have 4 years of working experience in web technologies and I have a strong working knowledge of every possible framework which is in the market. In terms of client-end I know: Javascript, PHP, Python, Sof More

$1250 USD in 29 days
(91 Reviews)

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the More

$500 USD in 7 days
(147 Reviews)

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields More

$500 USD in 4 days
(137 Reviews)

Hi there, Being an expert in Python and matlab programming I can do this project for you. please leave a message on my chat so we can discuss the budget and deadline of the project. Also, have a look at my profile for More

$750 USD in 3 days
(22 Reviews)

Hi. I will develop the web crawler that has GUI and key features using python. As you want, i will create 3 options I have over 5 years of experience with python. I have already completed several scraping projects on More

$750 USD in 5 days
(47 Reviews)

Good day! I'll develop you a fast and powerful web crawler to extract data from Google SERP. Important key features; 1. With a user interface protected by user and password where you can manage your campai More

$500 USD in 7 days
(13 Reviews)

Hi there, Greeting!! I have more than 10 years of experience in Web & App development and multiple technologies. I am committed to providing you high quality services. My goal is to satisfy clients by providing high q More

$750 USD in 7 days
(39 Reviews)

Hello I'm familiar Page Scrapping with Phantom JS and php script. I have some examples. I read your description and i can start right now. Please discuss details on chat, Thanks

$700 USD in 7 days
(6 Reviews)

Hi there, After reading your offer, this looks like a perfect fit for my skill sets so May I discuss with you for further details about this project? I'm a web developer with over 7 years experience and very strong i More

$250 USD in 4 days
(8 Reviews)

hello sir , I am a full time freelancer and have 10 years of experience in web and window development in c#, javascript,sql server . and Android mobile development i can do work as per your requirement . More

$500 USD in 7 days
(2 Reviews)

Hello, I checked out your requirements.I have 5+ years of working experience and Expertise in IOS , Android , Web Development, PHP, JavaScript, NodeJS, HTML,C++ ,Wordpress, CSS, RestAPI, Third-Party integration. I a More

$500 USD in 7 days
(2 Reviews)

Hi there. I have carefully read your job description with great interests. I have experience with python, django,selenium and bs4(beautifulSoup) for about 5 years. Please visit my past references here. -http://cars More

$500 USD in 7 days
(4 Reviews)

Hi there! I have read your description carefully and I am confident I can full fill your requirement. I provide my skills in hopes of working with you. I am a web scraper and have 7+years of ample experienced in web s More

$500 USD in 7 days
(3 Reviews)

Hello Dear, Yes, I can work for you as per your project description with amazing exploration and experimental research abilities of mine organization is perfect for that you may require. My work fulfills as well as ke More

$600 USD in 10 days
(1 Review)

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide More

$500 USD in 7 days
(0 Reviews)

Hi I have read your post content carefully and can develop your python project as my experience. That's why I have rich experience with Python. I am a new freelancer on, but I can provide you robust and More

$500 USD in 7 days
(0 Reviews)

I READ YOUR POST I CAN DO IT FOR YOU Hello I am a senior developer. I want to do it for you if you want. we can discuss more about this project. if you can send a reply. I waiting for you best response. let me discuss More

$500 USD in 7 days
(0 Reviews)

Nice to meet you Hi I am automation expert If there is any website you need to scrape data from (no matter the format, what blocking tools there are). I can deliver the data. If you do the same thing on the internet More

$637 USD in 9 days
(0 Reviews)