Are you looking forward to scraping locational data from Google Map? Then come in now and discover how to use Python and Selenium to get it done easily with a few lines of code.
Google Map is arguably the most popular and used map on the Internet by regular Internet users. The service has been designed in such a way that the data on it is user-generated, and you can find tonnes of user-generated data, including location details of a business, reviews added by customers of a business, and the end rating.
For many people, finding out specific businesses and services to use for unfamiliar locations starts from Google Map as most serious businesses are depicted on the map with some adding information of their service to intending customers that will be looking them up on the map.
The enormous amount of location and business listing data available on Google Map has made it a great source of data for lead generators and review researchers, among others. The data, even though publicly available, has some restrictions attached to them – automated access for the purpose of collecting the data is not allowed.
Unfortunately, Google does not provide a free method you can use to collect the data. They do have a Place API with a limited free version that is not useful to scrapers that need to scrape a whole lot of listings. This has made web scraping the method of choice for collecting data from Google Maps.
In this article, we would be discussing how to scrape Google Map using Python and Selenium. This is to enable those with coding knowledge to develop their own custom Google Map scraper. We would also be providing you recommendations on already-map scrapers in the market that you can use to scrape Google Map even without coding skills.
Google Map Scraping – an Overview
Scraping is generally considered legal, but that does not mean you will find it easy. A lot of the data available on Google Map of interest to web scrapers is user generated. However, Google Map would not allow you to scrape data from its platform without a fight, and only when you can either evade the fight or even win it would you have unrestricted access to the data on the platform.
My first attempt at scraping Google Map was a failure – JavaScript and their anti-scraping systems put me off. I had to learn how to scrape Google Maps and apply the knowledge before I was able to successfully extract the data I wanted.
Google Map scraping requires JavaScript rendering, and as such, only HTTP libraries that can render JavaScript will work, which means that Scrapy, Requests, and other HTTP requests that do not load and interpret JavaScript will not work.
For Python programmers, Selenium is the tool for the job as it is used to automate web browsers, and you can get it to open and load all of the data on a Google Map page in other to extract them.
Aside from rendering JavaScript, you will need proxies to hide the IP footprints associated with your web requests, as using one IP address will get that IP address banned if you end too many requests within a short period of time.
How to Scrape Google Map Using Python and Selenium
As a coder, you will not have to need to use already-made Google Map scrapers as you can develop a custom one that will meet your need. Most already-made Google Map scrapers were developed to be used as standalone tools that either come with unnecessary features or do not come with all of the features you will like. If you’re a coder, then you can develop a Google Map scraper in no time, as there are libraries and frameworks you can use to develop a Google Map scraper tailored to your need.
Google Map is an interactive service, and as such, you cannot use regular scraping tools that do not render JavaScript as the Google Map website depends heavily on JavaScript to generate and output results. For this reason, you cannot use the duo of Requests and Beautifulsoup – Scrapy also will not work as it does not support JavaScript.
For developing Google Map scrapers using the Python programming language, you will need Selenium. This tool automates web browsers to carry out specific tasks. With this tool, you can launch Google Chrome, load the Google Map website, search for specific businesses, click on them, copy the displayed information.
While Selenium will do the work, there’s still the risk of getting detected and banned. For this reason, you will need to incorporate some anti-scraping techniques, such as using rotating proxies that will change your IP after either every request or after a short period of time.
Also, you will need to change the user agent, set random timing within requests, and mimic human actions such as typing and, clicking, and moving of cursor. In all, you will want to be as natural as possible and avoid getting your requests interpreted as originating from a bot.
-
Install Python and Selenium
To use Selenium and Python, you will need both installed and then download a Chrome driver. Selenium requires a Chrome browser to function. To install Python, visit the official Python website and download the latest version compatible with your computer. To install Selenium, go to the command line and enter the command below to download Selenium.
pip install selenium
With Python and Selenium installed, you will need to install a browser driver that will serve as an interface between Selenium and your browser. Selenium has support for popular web browsers. In this guide, we would be using Google Chrome. Visit the Google Chrome driver page to download the version for your browser. It will be in a zip file. Unzip it and copy the content to the folder you will be creating the script in. It is advisable to read this Selenium Python guide to learn how to use Selenium for Python programming.
-
Code Sample for Scraping Google Map Business Listing
To explain how to code a Google Map scraper using Python and Selenium, I will be showing you a Minimum Viable Product (MVP) Google Map Scraper developed using Selenium and Python. The tool is very simple and contains just one class. If you provide it a list of URLs linking to businesses of interest on Google Map, it will provide you the detail of each business as a dictionary in a list.
The tool is quite simple and meant only for showing how it can be done. Anti-scraping techniques were not implemented, and we did not also implement exception handling.
Below is the code of the simple Google Map scraper. Create a python file and paste the code below. Save the file in the same folder you placed the Chrome driver file you downloaded and unzipped earlier.
from selenium import webdriver
class GoogleMapScraper:
def __init__(self):
self.PATH = "chromedriver.exe"
self.driver = webdriver.Chrome(self.PATH)
self.business_list = []
self.business_info = {}
self.business_info["name"] = "NA"
self.business_info["rating"] = "NA"
self.business_info["reviews_count"] = "NA"
self.business_info["address"] = "NA"
self.business_info["contact"] = "NA"
self.business_info["website"] = "NA"
def get_business_info(self, url):
self.driver.get(url)
# Parse data out of the page
self.business_info["name"] = self.driver.find_element_by_class_name("x3AX1-LfntMc-header-title-title").text
self.business_info["rating"] = self.driver.find_element_by_class_name("aMPvhf-fI6EEc-KVuj8d").text
self.business_info["reviews_count"] = self.driver.find_element_by_class_name("widget-pane-link").text
self.business_info["address"] = self.driver.find_elements_by_class_name("QSFF4-text")[0].text
self.business_info["contact"] = self.driver.find_elements_by_class_name("QSFF4-text")[1].text
self.business_info["website"] = self.driver.find_elements_by_class_name("QSFF4-text")[2].text
#add business info to business list
self.business_list.append(self.business_info)
urls = ["https://goo.gl/maps/Kfpy778TpLxj3Nyw6",]
BusinessScraper = GoogleMapScraper()
for url in urls:
BusinessScraper.get_business_info(url)
print(BusinessScraper.business_info)
You may be like to read,
- Python Web Scraping Libraries and Framework
- How to Scrape Tweets From Twitter with Python
- How to Scrape YouTube video, comments with Python
- How to extract data from Instagram with Python
Best Google Map Scrapers in the Market
The above guide is for coders who are interested in developing their own custom Google Map scraper. If you’re not a coder, you do not have anything to do with the guide above; this section is for you. There are some web scrapers designed for scraping without the need to code.
These already-made web scrapers are suitable for non-coders. There are a good number of them in the market, but we would be discussing 5 popular options that you can use for scraping Google Map.
Octoparse
- Pricing: Starts at $75 per month
- Free Trials: 14 days of free trial with limitations
- Data Output Format: CSV, Excel, JSON, MySQL, SQLServer
- Supported Platform: Cloud, Desktop
Octoparse is arguably one of the best Google Map scrapers in the market. It was not specifically developed as a Google Map scraper; it is a general scraping tool that you can use for scraping all kinds of websites, including both old static websites to modern interactive and Ajaxified websites. Octoparse has anti-bot techniques implemented to evade detection and ban.
Octoparse has support for scraping Google Map, and there’s a guide on how to use it to scrape business listing on the Octoparse blog – click here to read the guide. This software is available as both downloadable software and a web service. This tool is paid, but as a first-time user, you can use it for free for 14 days.
ParseHub
- Pricing: Free
- Free Trials: Free – advance features come at an extra cost
- Data Output Format: Excel, JSON,
- Supported Platform: Cloud, Desktop
Another web scraper you can use for scraping data from Google Map and other locational maps is ParseHub. This software is quite popular among non-techies that want to scrape data from the Internet. It also has API support for developers that do not want to reinvent the wheel by developing a custom scraper.
If you do not have a budget, ParseHub is the best tool for the job as it has a free tier you can use. However, not all features are available in the free tier, and you’re bound to hit lots of restrictions. For unrestricted access to the full potentials of ParseHub as a Google Map Scraper, go for their paid version. Here’s a guide on how to use ParseHub for scraping Google Maps.
Apify Google Map Crawler
- Pricing: Starts at $49 per month for 100 Actor compute units
- Free Trials: Starter plan comes with 10 Actor compute units
- Data Output Format: JSON
- Supported OS: cloud-based – accessed via API
As a developer, the Google Place API is available to you, but you do not have a lot of freedom if you use it. If you’re looking for a more robust API solution to scraping reviews, business contact details, and photos, among others, from Google Map, then the API Google Map Crawler is the tool for you.
This software has proven to be one of the best you can use as a developer if you do not want to deal with anti-scraping systems. The developer of this API solution provides easy-to-understand documentation and guidance on how to use this software for collecting data from Google Maps.
WebHarvy
- Pricing: Starts at $139 for a single user license
- Free Trials: Not available
- Data Output Format: TXT, CSV, Excel, JSON, XML. TSV, etc.
- Supported Platforms: Desktop
WebHarvy has proven itself to be one of the best web scrapers for non-coders. It is a general-purpose web scraping tool that you can use to scrape all kinds of data from any website on the Internet. It is a visual web scraper, and all you need is to use the point and click interface to select items of interest.
One feature you will come to like about WebHarvy is that it has support for regular expression, which makes it perfect for scraping content buried withintext strings. You can use WebHarvy to scrape Google Map business listings and reviews. Here’s a comprehensive guide on how to use WebHarvy for scraping Google Maps.
- WebHarvy Proxy 101: How to Integrate Proxies with WebHarvy
- Best Web Scraping Tools – Ultimate Web Scraper List!
ScrapeStorm
- Pricing: Starts at $49.99 per month
- Free Trials: Starter plan is free – comes with limitations
- Data Output Format: TXT, CSV, Excel, JSON, MySQL, Google Sheets, etc.
- Supported Platforms: Desktop
Last on the list of web scrapers you can use to scrape Google Map data is ScrapeStorm, a powerful web scraper developed by an ex-Google crawler team. This web scraper is a visual web scraper that provides you a point and clicks interface to select items of interest on the map. It can take care of pagination and every other workflow involved in web scraping.
The team behind the development of this project is experienced, and as such, you will not have to worry about blocks. One feature ScrapeStorm has that others above lack are the variety of data output format supported – ScrapeStorm has support for many data output formats.
Conclusion
Scraping Google Maps is not as difficult as you think, provided you are equipped with the right knowledge and skill to access the service in an automated manner without detection and parse out the required data.
You will agree with me that we have discussed how to get this done. Not being a coder is no longer a barrier – already-made web scrapers are now available that you can use, and some of the best are listed below.
