How to Configure Proxies for the Requests Library [Python Requests Proxy Setup]

The Python third-party library, Requests does have support for proxies. The article below has been written to provide you guidance on how to set up Requests so that their web requests are routed via any proxy server of your choice.

Python Requests Proxy Setup

Proxies are incredibly important for you as a developer looking to automate your workflow and activities online. They are quite helpful in the area of security and privacy as the web has turned into a marketplace for tracking users and their activities. Aside from tracking, there is also the issue of censorship, IP blockage, and geo-targeting that proxies would help you solve.

One good thing about proxies is that most developer tools that have network/Internet access do have support for it including the Python Requests library which is quite popular among Python developers looking to create bots such as web scrapers, form fillers, and purchase bots, among others.

In this article, you will be shown how to set up proxies when using the Python Requests library to send web requests. You will also get recommendations on the best proxies to use as not using the right proxies could ruin your project for you. Before showing you how to configure proxies in the Requests, let take a look at an overview of the Requests library.


Overview of Requests Library

Requests dubbed HTTP for Humans a simple HTTP library for Python programming. This library does not come installed in the Python standard library. There is an HTTP package in the standard library known urllib which among other things, has support for sending HTTP requests.

However, not many Python developers love to make use of it. This is because it is messy, requires more lines of code and error handling which is not convenient for developers. For this reason, a good number of HTTP libraries were built. The Requests library is the most popular among all of the third-party libraries for sending HTTP requests in python. Interestingly, it is built on top of the standard library urllib package.

YouTube video

Requests take away all of the complexities and difficulties you will have if you were to use urllib by providing you an easier to use API that requires fewer lines of code, it also throws better and easier to understand exceptions and it is easy to code in and manage when compared to urllib or any of the other HTTP libraries.

Because of this, it has become the HTTP library of choice for most Python developers especially those coding bots for web tasks automation such as web scrapers and crawlers, product checkout bots, and form submission bots among others. Requests have support for the popular HTTP methods which include POST, GET, PUT, and DELETE. It also has other methods such as head and PATCH.


Procedures for Requests Library Installation

Procedures for Requests Library Installation

Requests do not come preinstalled in the standard Python distribution. For this reason, you will need to install it before you can use it. However, before installing it, you will need to have Python installed and pip.

  • Python Installation

For most computers, Python comes installed but the Python installed is the outed Python 2. If that is what you have installed, you will need to download and install the current Python 3 version. You can download the latest version of Python from the official download page — make sure you install the one appropriate for your computer.

  • PIP Installation

Pip is a package manager for python which you can use for installing Requests and other Python packages and modules. Aside from Python, you also need to have pip installed before you can use Requests. If you already have python 3 installed, you need to verify if you have pip installed. You can verify that with the command below

pip —version

If it is not installed, then you will need to get it installed. To install pip, download the get-pip.py file from the official link. Copy the get-pip.py into a folder easily accessible. Then using the command prompt or terminal, navigate to the folder. If you re using either Linux or macOS, enter the below command.

python get-pip.py

If you are using Windows, run the below command to install pip.

py get-pip.py

With the command ran successfully and pip installed, you can go to the next step.

  • Requests Installation

All you need to install Requests is the pip command below.

python -m pip install requests

Request Hello World Program

python PIP Installation

As with most programming tutorials, we would be providing you with a Hello World program. In this case, it is going to be in the form of a bot that visits a website and crawls all of its pages, adding new URLs to the list of URLs to be crawled and repeating the same until there is no new URL to be crawled.

import requests

from bs4 import BeautifulSoup

class Crawler:

    def __init__(self, start_url):

        self.start_url = start_url

        self.crawled_urls = []

        self.unvisited_urls = []

        self.start()

    def crawl(self, url):

        print("crawling " + url)

        fetched_page = requests.get(url)

        self.crawled_urls.append(url)

        soup = BeautifulSoup(fetched_page.text, "html.parser")

        urls = []

        for link in soup.find_all('a'):

            print(link.get('href'))

        for x in urls:

            if x in self.crawled_urls:

                pass

            else:

                self.unvisited_urls.append(x)

    def start(self):

        self.crawl(self.start_url)

        for url in self.unvisited_urls:

            self.crawl(url)

x = Crawler("https://amazon.com")

If you try running the code above on a website that has an anti-spam system such as the popular websites on the Internet today, you will get blocked after a while. This is because of the too many requests sent which would quickly make your crawler identifiable as a bot.

To avoid getting blocked, proxies are some of the effective tools to use. Basically, when configuring proxies, you may or may not be required to authenticate with a username and password. For this reason, we would be showing you how to use proxies with or without username and password authentication.


Configuring Proxies with No Authentication

Configuring Proxies with No Authentication

Some proxies support IP authentication which requires no username and password for authentication. For these proxies, all you need to set up is the proxy address and port. Other proxies in this category are open proxies AKA free proxies that are open to everyone to use and as such, there is no need or requiring authentication.

If the proxies you have access to do not require a username and password authentication, setting them up is the easiest and requires fewer lines of code. All you need to do is create a dictionary with the proxies and add it as an argument to the get method as done below.

import requests

proxies = {

   'http': 'http://proxy.example.com:8080',

   'https': 'http://secureproxy.example.com:8090',

}

url = 'http://mywebsite.com/example'

response = requests.post(url, proxies=proxies)

Python Requests Proxy Authentication

Proxies with Username and Password Authentication

Proxies that require a username and password authentication are configured in a different way. However, the difference is not that much.

All that is required is for the syntax above to be tweaked and changed in other to accommodate the username and password for authentication. Below is a code showing how to configure proxies with username and password authentication.

proxies = { 'https' : 'https://user:[email protected]:port' }

r = requests.get('https://url', proxies=proxies)

How to Configure Proxy List for IP Rotation in Requests

If you have access to a rotating proxy that changes IP address frequently as in the case of most residential proxies, you will not need to worry about rotating IP address. However, the proxies are private proxies or free proxies with individual proxy addresses and ports, then you will need to create a logic for IP rotation.

import requests

from itertools import cycle

list_proxy = ['socks5://Username:[email protected]:20000',

              'socks5://Username:[email protected]:20000',

              'socks5://Username:[email protected]:20000',

               'socks5://Username:[email protected]:20000',

              ]

proxy_cycle = cycle(list_proxy)

# Prime the pump

proxy = next(proxy_cycle)

for i in range(1, 10):

    proxy = next(proxy_cycle)

    print(proxy)

    proxies = {

      "http": proxy,

      "https":proxy

    }

    r = requests.get(url='https://ident.me/', proxies=proxies)

    print(r.text)
YouTube video

Best Proxies for Python

From the above, you already know how to set up proxies. But do you know the best proxies to use? If you already have a proxy server to use in mind, then this section is not for you.

However, if you are looking for a recommendation on the best proxy server to use in your custom Python script, then you can read this section. We would be suggestion top 3 proxy providers you can buy residential proxies from to use.


Bright Data

Bright Data - Luminati

  • IP Pool Size: Over 72 million IPs
  • Locations: All countries and major cities around the world
  • Concurrency Allowed: Unlimited
  • Bandwidth Allowed: Starts from 20GB
  • Cost: Starts from $300 monthly for 20GB

Bright Data for data Collector

Bright Data is arguably the best proxy provider in the market right now. It has proxies in all of the proxy categories — residential, datacenter, and mobile. However, it residential proxy network is the most popular of its proxies which are some of the best in the market right now. They are rotating proxies that change your IP address frequently.

You can either opt in to change it after every request or get it changed after a specific period of time as in the case of their session proxies. The network has one of the largest proxy pools in the market with over 72 million IP addresses sourced from all countries and major cities across the globe. Their proxies are compatible with most sites.


Soax

Soax Logo for all proxy

  • IP Pool Size: Over 5 million IPs
  • Locations: Over 100 countries
  • Concurrency Allowed: Unlimited
  • Bandwidth Allowed: Starts from 5GB
  • Cost: Starts from $75 monthly for 5GB

Soax Overview

Soax is also one of the best proxy providers you can use their proxies in your python script. The proxy provider claims to own one of the cleanest proxy pools in the market as it regularly checks its proxy pool and removes bad IPs. Soax has got a proxy pool with over 5 million residential IP addresses sources from many countries across the globe.

The proxies you get from this provider are rotating proxies. One thing you will come to like about Soax is its pricing, which is considered pocket-friendly as you can get started with only $75 as opposed to Bright Data that you need $300 to get started.


Smartproxy

smartproxy image logo

  • IP Pool Size: Over 40 million IPs
  • Locations: 195 countries
  • Concurrency Allowed: Unlimited
  • Bandwidth Allowed: Starts from 5GB
  • Cost: Starts from $75 monthly for 5GB

smartproxy

Smartproxy is a premium proxy provider that is known for its speed. Smartproxy has one of the fastest speeds in the residential proxy market — even faster than Bright Data. However, the reason we included it as a recommended provider is not just for its speed. It also has one of the largest proxy pools with over 40 million IP addresses from about 195 countries across the globe.

Smartproxy is also one of the most affordable providers. Just like the other two above, it pricing is based on bandwidth usage. However, it can be argued that Soax copied Smartproxy pricing considering they changed their pricing to that of Smartproxy — we have been following both providers for a long time now.

Conclusion

Requests is one of the libraries that has been developed to make sending of web requests easy for Python developers. Looking at the above, you will see that setting proxies is also easy and requires just a little modification to add support for sending your requests via proxies. If you d not have a proxy you want to use, we suggest you make a purchase from one of the providers we recommend above.