Scraping Yellow Pages is incredibly easy. To learn how to scrape contact details by coding a Yellow Page scraper or using an already-made tool, keep reading the below article.
Yellow Pages Scraper and How to Scrape Yellow Pages
Cold marketing is still one of the major ways businesses search for prospective customers and clients. With the right procedure, you could convert a total stranger into a loyal customer and make money from him. But how do you know who to contact and for what? Traditionally, marketers look through business directories in papers or specialized books such as Yellow Pages in search of contact details of prospective businesses.
Usually, the businesses listed didn’t get listed to get sold to but listed to gain more exposure and customers. However, the world has gone digital, and business directories in paper format are fading away, giving way to e-business directories.
The introduction of e-business directories such as Yellow Pages and Yelp has made searching for businesses easy and swift. Interestingly, it also opens up opportunities for marketers to quickly collect contact details and other business information from these directories.
Unfortunately, it does not come easy. Business directory websites wouldn’t willingly hand over their contact with any random person on the Internet. You will have to extract it yourself, and when you need to extract many listings, doing so manually won’t be effective and efficient. You will need to do so via web scraping.
Yellow Pages Scraping – an Overview
How do Yellow Pages scraping works?
A computer program known as a web scraper is used to visit the pages with the business listing and then extract listing data of interest in an automated manner. The web scraper downloads the HTML of the page, parse out required data, and save it in an accessible format. That’s how simple it is, scraping Yellow Pages fundamentally. Unfortunately, the process is not as easy and straightforward as mentioned. Yellow Pages do not like being scraped, and as such, they put up a defense in the form of anti-scraping systems to discourage scraping.
However, the defense is only effective for the non-techies that do not know how to bypass them. It turns out that you could easily scrape Yellow Pages even with the anti-scraping systems. All you need to do is use rotating proxies to make IP tracking and blocking useless in other to exceed the request limit and use Captcha solver in case you are forced to solve Captchas.
When it comes to the choice of web scraper for scraping Yellow pages, you can either develop one for yourself or use an already-made one in the market. Generally, only coders can develop a scraper as developing one requires coding skills since they are written in computer programming languages.
If you do not know how to code, then the best option for you is to make use of an existing scraper. In this article, we will be showing you how to code your own Yellow Pages scraper and all you need to know. We will also be recommending some of the best Yellow Pages scrapers already in the market for you to use.
How to Scrape Yellow Pages Using Python, Requests, and Beautifulsoup
As a coder, you will want to build your own Yellow Pages scraper and incorporate features you like. Yellow Pages isn’t a difficult website to scrape. While it utilizes JavaScript, you can actually scrape data even without JavaScript enabled.
And for this reason, you will not require any tool that will help you execute JavaScript and introduce complexity. You can use any programming language of your choice to code a Yellow Page scraper. In this example, we will be making use of Python as it is arguably the most popular language used in coding web scraping bots.
With language chosen, we move to the next step of choosing libraries to use. Our Yellow Pages Scraper is going to be simple and straightforward – a minimum viable product and a proof of concept, to be precise. We will be using the Requests library for sending HTTP requests downloading HTML for the Yellow Pages.
Beautifulsoup will be used for parsing. For tutorials, proxies are not required as we will only be sending a few requests. However, I am surfing from a country that does not have access right to the Yp.com service, and as such, I can’t access Yellow Pages from my location without a proxy server. The reason I stated this is because you might be in the same situation as mine. However, I will be taken the proxy setup part from the code.
Below is the code for a very simple Yellow Pages Scraper that extracts business details from a specific URL. Take note; we did not incorporate any anti-bot bypass technique for simplicity.
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.yellowpages.com/los-angeles-ca/dentists'
response= requests.get(url, headers=headers)
soup= BeautifulSoup(response.content,'lxml')
for item in soup.select('.v-card'):
print('----------------------------------------')
print(item.select('.business-name')[0].get_text())
print(item.select('.rating div')[0]['class'])
print(item.select('.rating div span')[0].get_text())
print(item.select('.phone')[0].get_text())
print(item.select('.adr')[0].get_text())
print('----------------------------------------')
print('')
- Scrapy Vs. Beautifulsoup Vs. Selenium for Web Scraping
- How to extract data from Instagram
- How to Build a Simple Web Scraper with Python
Best Yellow Pages Scrapers in the Market
The above section is for coders. If you are not a coder, you have nothing to worry about. There are web scrapers you can use to scrape Yellow Pages at affordable prices.
One thing you will come to like about these tools is that you will be using a visual tool to train the tools on the data to scrape. There are a good number of web scrapers you can use for scraping Yellow Pages. Instead of going through all of them, we will be going through some of the best. Below are 5 web scrapers you can use to scrape Yellow Pages.
Apify Yellow Pages Scraper
- Pricing: Starts at $49 per month
- Free Trials: Fully functional free account with $5 credit every month
- Data Output Format: JSON, CSV, Excel, XML, HTML, RSS
- Supported Platform: Cloud, Desktop
The Apify web scraping and automation platform has a fully customizable Yellow Pages Scraper that you can use to extract addresses, phone numbers, categories, ratings, and names from Yellow Pages.
You can scrape by using a combination of search query and search location, or you can specify a URL where the tool should start scraping. You can set the maximum number of pages to be scraped and the scraper can be scheduled to run with different parameters as often as you like.
You can choose to run the Yellow Pages Scraper on the cloud based Apify platform and take advantage of the integrated proxy service and fast servers, or you can run it locally on your own system.
ParseHub
- Pricing: Free
- Free Trials: Free – advance features come at an extra cost
- Data Output Format: Excel, JSON,
- Supported Platform: Cloud, Desktop
You do not have a budget but want to scrape business contact details from Yellow Pages? Then I will suggest ParseHub to you. It is a general web scraping tool that you can use to scrape all kinds of websites, including Yellow Pages.
There is even a tutorial on the ParseHub Blog that describes how to use the ParseHub scraper to extract business names and contact details from Yellow pages – you can check out the blog post here. One thing you will come to like about ParseHub is that it has a free tier you can use without paying a dime. With ParseHub, you can not only scrape but can also export the data in many formats.
With this tool, you will not have to write a line of code. All you need to do is make use of the point and click interface provided to specify the data point of interest, and the tool will take care of the rest.
Octoparse
- Pricing: Starts at $75 per month
- Free Trials: 14 days of free trial with limitations
- Data Output Format: CSV, Excel, JSON, MySQL, SQLServer
- Supported Platform: Cloud, Desktop
Octoparse is another general-purpose web scraping tool that you can use to scrape Yellow Pages. Unlike ParseHub that has a free tier with limitations that you can use anytime, Octoparse, on the other hand, only has a free trial option. It allows users to use it for 2-weeks free, after which you are expected to make payment.
Octoparse also provides a step by step guide on how to scrape Yellow Pages. The Octoparse scraping solution has a cloud scraping platform that you can use to scrape Yellow Pages 24 hours a day, 7 days a week. It also has support for schedule scraping, where you set up the scraper to run at intervals.
Other features of Octoparse includes support for automatic IP address rotation, exporting of scraped data as CSV or Excel file, accessible via API, or could be saved in databases. If you do not want to interact with the tool directly, they have got a professional data service that could help you out.
Yellow Scrape
- Pricing: Starts at $75
- Free Trials: Available
- Data Output Format: CSV
- Supported Platforms: Desktop
Unlike the above scrapers that are general scrapers, Yellow Scraper is not a general scraper and has been developed specifically for Yellow Pages scraping. It is the most specialized tool on the list and available only on Windows.
If you want to use it on Mac or another operating system, you will have to run it in a virtual machine, which will be an extra headache. With Yellow Scrape, you can scrape thousands of businesses in minutes. Yellow Scrape will help you extract business name, office address, and phone number, including website, emails, social properties, and contact names.
One feature that comes with Yellow Scrape that you will come to like is it email verifier that you can use to verify if the emails for the business you scraped are working. You can also use the software to test for mobile responsiveness of business websites.
ScrapeStorm
- Pricing: Starts at $49.99 per month
- Free Trials: Starter plan is free – comes with limitations
- Data Output Format: TXT, CSV, Excel, JSON, MySQL, Google Sheets, etc.
- Supported Platforms: Desktop
ScrapeStorm is a visual web scraping software that you can use to scrape Yellow Pages. One thing you will come to like about ScrapeStorm is that it is an Artificial Intelligence-based web scraping tool. You do not have to specify certain details you want to scrape as it can automatically detect them. ScrapeStorm is built by an ex-Google crawler team, and as such, the team behind it is experienced.
The visual click operations required are minimal, and the speed of execution is very fast. ScrapeStorm has support for multiple data export formats – and one of the best support in the market.
ScrapeStorm is not your regular scraping tool; it is built to serve as a performance beast and serve enterprises. It has a cloud scraping platform that makes it convenient as you can access it from any computer with Internet access. However, it has the software you can install on your computer and has support for Windows, Mac, and Linux.
WebHarvy
- Pricing: Starts at $139 for a single user license
- Free Trials: Not available
- Data Output Format: TXT, CSV, Excel, JSON, XML. TSV, etc.
- Supported Platforms: Desktop
Last on our list of web scrapers that can be used for scraping Yellow Pages is WebHarvy. WebHarvy is an intuitive web scraper that can extract text, URL, emails, and images, among others, from web pages. WebHarvy is perfect for scraping Yellow Pages.
To learn how to use WebHarvy to scrape Yellow Pages, click here. WebHarvy has support for proxy usage, but you are required to set up proxies yourself. It also has support for a scheduler, which makes it good for scheduling scraping tasks to even automatically without you initiating it.
WebHarvy is not a free tool. It comes with a price tag but has a trial option to use. This tool comes with many advanced features, including support for intelligent pattern detection, scrape by keyword, regular expression support, browser automation, and category scraping, among others.
- Yelp Scraper 101: How to Extract Data from Yelp pages
- How to Scrape Data from Linkedin Using Proxies
- Email Extractor: Email scraping services and Software
Conclusion
Looking at the above, you can see that regardless of your coding knowledge, you can actually generate tonnes of lead from Yellow Pages and use it for your cold marketing. For coders, creating one will give them the opportunity of building to taste.
However, even without coding skills, you can use the web scrapers we discussed above for your Yellow Pages scraping tasks. While at it, you need to know that you are doing nothing illegal by scraping since the data is publicly available. However, what you plan on using the scraped data for will make it illegal or not. It is best you contact a lawyer to know where you stand before going ahead.
