How to Use Python Requests Retry Effectively

Modern web APIs and scraping projects frequently encounter HTTP request failures. A server might be momentarily down, a network hiccup could occur, or a target website may impose rate limits.

In these cases, having your Python script automatically retry failed requests can save the day. The requests library makes sending HTTP requests in Python easy, but it doesn’t magically eliminate errors like timeouts or connection drops.

In this guide, we’ll explore how to use Python requests retry mechanisms effectively: from leveraging requests’ built-in features to writing custom retry loops with exponential backoff, handling special HTTP codes, using proxies for anti-blocking, and more.

Common causes of request failures

Before implementing retries, it’s important to understand why requests fail in the first place. Common causes include network issues, server problems, and client-side errors:

• Network issues

Temporary network blips can cause connection errors or timeouts. For example, a poor internet connection or DNS issues might prevent your request from reaching the server at all.

• Server-side errors

HTTP codes in the 5xx range (500-599) indicate the server encountered an error fulfilling the request. For instance, a 500 Internal Server Error or a 502 Bad Gateway suggests something went wrong on the server’s end. These are often temporary; the server might recover after a short time.

• Client errors and rate limiting

HTTP 4xx codes generally indicate a problem with the request. Some, like 404 Not Found or 401 Unauthorized, are permanent unless you change the request (correct the URL or credentials). Others are more transient.

A 429 Too Many Requests means you’ve hit a rate limit, as the server is asking you to slow down. Similarly, a 403 Forbidden might occur if the server blocked your access (for example, due to scraping protection).

In summary, retry failed requests only for issues likely to be temporary. Network hiccups, exceptions like timeouts, and 5xx server errors are prime candidates for retries. But permanent errors (400-level issues other than rate limiting) should usually not be retried blindly.

Understanding the cause of failure guides whether a retry is worthwhile or if you need a different fix (such as correcting a bad URL or including proper authentication).

Using HTTPAdapter to retry Python requests

Python’s requests library doesn’t retry failed requests by default, but it provides tools to implement retries. Under the hood, requests uses the urllib3 HTTP client, which includes a Retry class to handle retry logic.

The key is to use a session object with a custom HTTPAdapter. The HTTPAdapter allows you to configure a retry strategy that the session will use for all requests.

How built-in retries work in requests

Out of the box, the requests library will not automatically retry failed requests (aside from maybe a few specific scenarios like redirected requests). This is by design, as the library leaves retry behavior up to the developer.

The good news is requests lets you plug in a Retry configuration via its adapter mechanism. The requests.adapters.HTTPAdapter can be configured with a max_retries parameter, which accepts a Retry object from urllib3. By attaching this adapter to a session, you enable automatic retries for requests made through that session.

The Retry class from urllib3 provides flexible retry behavior. You can specify: how many total retries to allow, which HTTP codes should trigger a retry, which request methods to retry, and how to wait between retries (e.g., using a backoff algorithm).

Importantly, the Retry logic will only apply to requests made via the session that has the adapter mounted. It won’t affect other usage of requests.get() or such unless those calls go through the configured session.

In the next sections, we’ll walk through an example of configuring an HTTPAdapter with Retry and discuss the key settings you can tune for effective retries.

Example: Configuring HTTPAdapter with Retry

Let’s set up a basic example using requests’ built-in adapter mechanism to automatically retry failed requests. We’ll use the HTTPAdapter class (from requests.adapters import HTTPAdapter) in combination with urllib3’s Retry class:

    	import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
# Define a retry strategy
retry_strategy = Retry(
    total=3,                   # total retry attempts (excluding the first request)
    backoff_factor=1,          # factor for delay between retries
    status_forcelist=[429, 500, 502, 503, 504],  # HTTP codes to retry
    allowed_methods=["HEAD", "GET", "OPTIONS"]   # methods to retry (None means any)
)

# Create an adapter with the retry strategy
adapter = HTTPAdapter(max_retries=retry_strategy)
# Create a session and mount the adapter to both HTTP and HTTPS
session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)

# Now use session to make requests
try:
    response = session.get("https://httpbin.org/status/503", timeout=5)
    print(f"Final status: {response.status_code}")
except Exception as e:
    print(f"Request failed: {e}")
  	

In this code, we configure a retry strategy that will attempt up to three retries (so up to four total tries counting the initial request) for certain HTTP codes. We’ve set status_forcelist to [429, 500, 502, 503, 504], meaning any response with those codes is considered retryable.

The backoff_factor=1 will enforce a delay between retries (we’ll explain the backoff calculation shortly). We mount our adapter to a session object, and then use session.get for the request.

Now, if the target URL responds with a 503 Service Unavailable, the session will automatically retry the request up to three times before giving up.

Notice we included a timeout=5 in the get() call. It’s a good practice to set a timeout on your requests, especially when using retries, so that each attempt doesn’t hang indefinitely.

Key parameters to tune

When using HTTPAdapter with a Retry object, you have several important parameters you can adjust for your use case:

• Total retries

This controls the total number of retries (not counting the initial request). In our example, we used total=3, which means the request can be attempted 3 additional times if it fails. Setting total=None would allow unlimited retries (not recommended), and total=0 would disable retries altogether.

• Status code filtering

The status_forcelist parameter is a list of HTTP codes that should trigger a retry. By default, urllib3’s Retry won’t retry on codes at all unless you specify this list.

• Allowed HTTP methods

By default, retries are only enabled for “safe” HTTP methods like GET, HEAD, OPTIONS, and PUT/Delete, which are considered idempotent. The idea is to avoid retrying non-idempotent requests (like POST) since resubmitting a form or purchase action could have side effects.

• Backoff settings

The backoff_factor is a crucial setting to avoid bombarding the server. This factor, combined with the retry count, determines how long to wait before the next attempt.

• Error handling behavior

By default, once the retry limit is exhausted, the requests session will raise an exception (a MaxRetryError from urllib3, which will be wrapped in a requests.exceptions.RetryError).

With these parameters, you can fine-tune how your retry strategy behaves. For instance, you might only want to retry on connection errors and not on HTTP errors, or vice versa - this can be configured by adjusting the Retry settings. The built-in approach with HTTPAdapter is convenient for many cases and keeps your code fairly concise once set up.

Custom retry logic & decorators

The built-in adapter approach is handy, but sometimes you need more customization than it provides. Perhaps you want to add logging, implement a special retry logic (like refreshing an authentication token on failure), or handle conditions that urllib3’s Retry doesn’t cover (for example, retrying based on the content of a successful response).

In such cases, writing your own retry loop or using a third-party library can be the way to go. Let’s look at a few approaches.

Try/Except-based retry loop

The simplest form of custom retry is to use a loop with a try/except block. This gives you full control over what errors to catch and how to respond. Here’s a basic pattern for a retry logic loop:

    	import requests
import time
max_retries = 3
for attempt in range(1, max_retries+1):
    try:
        response = requests.get(url, timeout=10)
        # Check if response is successful
        if response.status_code == 200:
            print("Success on attempt", attempt)
            break  # exit loop on success
        else:
            print(f"Received status {response.status_code}.")
            # Optionally, decide if this status should trigger a retry or not
    except requests.exceptions.Timeout as e:
        print("Request timed out, attempt", attempt)
        # We can retry on timeouts
    except requests.exceptions.RequestException as e:
        print("Request failed:", e)
        # Handle other exceptions (ConnectionError, etc.)
    # If we reach here, it means we need to retry
    if attempt < max_retries:
        time.sleep(2)  # wait 2 seconds before next attempt
    else:
        print("All retries failed.")
  	

In this snippet, we attempt the request up to three times. We explicitly catch a timeout exception (using requests.exceptions.Timeout) and treat it as a retriable condition. We also catch a general RequestException to cover other issues like connection errors.

After each failure, if we haven’t exhausted the retries, we wait a fixed two seconds before trying again.

Exponential backoff with jitter

When implementing custom retries, using an exponential backoff algorithm is a best practice. Rather than sleeping for a fixed interval between attempts, you start with a small delay and increase it exponentially with each retry. This reduces the load on the server and avoids a thundering herd of immediate retries.

Adding jitter means adding a bit of randomness to those delay values. Jitter helps when many clients or threads are retrying at once, as it staggers the retry times so that they don’t all hit the server again simultaneously.

Here’s how you might generate a list of backoff delays with optional jitter:

    	import math, random
def make_backoff_delays(base_delay, factor, max_retries, jitter=False):
    delays = []
    for n in range(max_retries):
        # Exponential backoff: factor^(n) * base_delay
        delay = (factor ** n) * base_delay
        if jitter:
            # random jitter between 0.5x and 1.5x of delay
            delay = delay * random.uniform(0.5, 1.5)
        delays.append(delay)
    return delays
# Example: 5 retries, base 1s, factor 2
print(make_backoff_delays(base_delay=1, factor=2, max_retries=5, jitter=True))
  	

If you run the above, it will output a sequence like [1.2, 2.1, 4.3, 8.5, 16.2] (the exact values will vary due to randomness). The general idea is that the wait grows longer with each attempt. Using exponential backoff prevents overwhelming the server and it’s a widely recommended practice instead of fixed delays, which can overload the server.

Retry using the tenacity library

If you prefer not to write retry loops from scratch, you can leverage the tenacity library, which is a powerful third-party package for retrying in Python.

To use tenacity, first install it (e.g., pip install tenacity). Then, you can apply its @retry decorator to a function that performs a request. For example:

    	from tenacity import retry, stop_after_attempt, wait_exponential
# Configure the retry decorator
@retry(stop=stop_after_attempt(4), wait=wait_exponential(multiplier=1, min=2, max=10))
def fetch_with_retry(url):
    return requests.get(url, timeout=5)
  	

In this snippet, @retry will retry the function up to four times (stop_after_attempt(4) means four attempts maximum) and it uses an exponential wait between tries. The wait_exponential configuration here starts with a two-second delay and doubles it each time (with a maximum of 10 seconds between retries).

When to prefer custom logic over HTTPAdapter

With both built-in HTTPAdapter retries and custom methods (including tenacity) available, you might wonder which to choose. Here are some considerations:

• Complex conditions

If you need to retry based on complex criteria, e.g., the content of a 200 OK response (as discussed earlier) or perhaps after performing some fix like refreshing an OAuth token, then custom logic is the way to go.

• Different actions per attempt

Sometimes you may want to change what you do on each retry. For example, on the second failure, you might switch to a different URL or adjust the request parameters.

• Integration and logging

If you need detailed logging, metrics, or integration with other parts of your system on each retry (like logging each attempt or notifying another service after X failures), a custom approach gives you that control.

• Specific exception handling

While urllib3 Retry will handle some low-level exceptions (like connection errors) automatically, you might want to explicitly handle others.

• Environment

In certain environments (such as asynchronous frameworks or when using aiohttp instead of requests), the HTTPAdapter approach isn’t applicable. Tenacity or manual retries can be adapted to those cases (e.g., using async sleeps).

Handling HTTP status codes like 429 and 503

Certain HTTP status codes deserve special attention when it comes to retries. Two common ones are 429 Too Many Requests and 503 Service Unavailable:

• 429 (too many requests)

This status code indicates the server is rate-limiting you. Essentially, you’ve sent too many requests in a short time. Many servers will include a Retry-After header in the 429 response, telling you how many seconds to wait before retrying.

Your retry logic should check for this. For example, if you get a 429, look at response.headers.get("Retry-After"). If it’s present and says "30", you should wait 30 seconds before the next attempt.

• 503 (service unavailable)

A 503 often means the server is temporarily unable to handle the request (server overload or maintenance). Like 429, it may come with a Retry-After header if the server expects to be available later. Treat this similarly by pausing retries for the specified duration.

For both 429 and 503 (and similar errors like 502 Bad Gateway or 504 Gateway Timeout), retrying can be effective if done with backoff and respect for server signals. Always avoid tight loops of rapid-fire retries on these responses. Using the Retry-After header is a best practice – if the server says “come back in 60 seconds”, listen to it.

Retrying Python requests with proxies and anti-blocking

When you’re scraping websites or accessing APIs extensively, you may encounter blocks or CAPTCHAs that simple retrying won’t solve. This is where residential proxies come into play. Using them means routing your HTTP requests through another server (or a pool of servers) so that the target website sees a different IP address.

Why proxies are important for retrying requests

Consider a scenario: you’ve sent several requests in a short time and suddenly start getting 403 Forbidden or 429 Too Many Requests responses. This could mean your IP has been flagged or temporarily banned by the server.

If you simply retry from the same IP, you’ll likely keep getting blocked (or even worsen the situation). By switching to a different proxy IP on retry, you might succeed on the next attempt because the server treats it as a new source. Essentially, proxy servers can provide a fresh identity.

Proxies are also useful for geo-restricted content. If a site only serves certain regions, a proxy from an allowed region would be necessary. In the context of retries, a common strategy in web scraping is: if a request fails due to an IP block, rotate to the next proxy and try again.

This increases the chance of success across multiple attempts, as you’re not hitting the server from the same address each time.

Adapting retry logic when using proxies

Integrating proxies into your retry logic requires a bit of adaptation. If you’re using the HTTPAdapter approach, you can specify proxies on the Session or per request using the proxies parameter. For instance:

    	proxies = {
    "http": "http://USERNAME:PASSWORD@proxy.server:port",
    "https": "http://USERNAME:PASSWORD@proxy.server:port"
}
response = session.get(url, proxies=proxies, timeout=10)
  	

You could set up a Session to use a default proxy, but often you may want to change the proxy with each retry attempt.

In a custom retry loop, you might do something like:

    	proxy_pool = ["proxy1.com:8000", "proxy2.com:8000", ...]  # list of proxy addresses
for attempt in range(max_retries):
    proxy = proxy_pool[attempt % len(proxy_pool)]
    try:
        resp = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=10)
        if resp.status_code == 200:
            return resp
        elif resp.status_code in [403, 429]:
            print("Got {}, switching proxy and retrying...".format(resp.status_code))
            # continue to next attempt with a new proxy
        else:
            break  # for other status codes, break out (non-retryable)
    except requests.exceptions.RequestException as e:
        # If a proxy itself fails (e.g., connection error), try next proxy
        print("Proxy {} failed, trying next proxy...".format(proxy))
  	

In this pseudo-code, if we get a 403 or 429, we assume it’s due to blocking and switch to the next proxy for the next iteration. If a proxy outright fails to connect, we catch the exception and also move on to the next one.

Implementing basic proxy rotation

A basic proxy rotation strategy is as shown above: maintain a list of proxy servers and pick a different one on each retry. You can randomize the order or cycle through them sequentially.

Make sure to test your proxies beforehand. Including a completely non-functional proxy in the rotation will just add extra retries that all fail. Some developers use a session object per proxy or a pool of sessions, each pre-configured with a proxy, to avoid rebuilding the session for each attempt.

Libraries and tools for proxy management

Managing proxy servers can get complex, especially if you need dozens or hundreds of IPs. There are several tools and services to help:

• Proxy rotation services

Companies like MarsProxies offer proxy networks and even APIs that automatically rotate proxies for you.

• Open source libraries

There are Python libraries, such as requests-ip-rotator, or scrapeops-python-proxy, that can plug into requests to handle rotations.

• Scrapy or other frameworks

If your project is on a larger scale, a web scraping framework like Scrapy has built-in support for proxy middleware and rotation.

• Rolling your own

For simpler needs, you might not need a fancy library. A combination of a list of proxies and the requests proxies parameter can go a long way.

Best practices for retrying requests

We’ve covered various ways to implement retries. To wrap up the how-to portion, let’s highlight some best practices when adding retry capability to your Python scripts:

• Limit the total retry cycles

It’s important to cap how many times you’ll retry. Don’t create an infinite loop that keeps retrying forever if a service is down: you’ll waste resources and possibly never recover.

• Use exponential backoff (avoid zero or fixed delays)

Always incorporate a backoff factor to gradually increase wait time between attempts. This prevents flooding a server.

• Set timeouts for each request

Each individual request should have a timeout (both connect and read, if applicable).

• Log retries and outcomes

When debugging or monitoring, it helps to have logging for retries.

• Avoid retrying non-idempotent operations (or handle with care)

As noted earlier, be cautious with retrying requests that aren’t safe to repeat. If you must retry a POST or PUT that could create duplicate actions, consider designing the server side to handle duplicates safely.

• Respect rate limits and backoff signals

Retries should not ignore what the server is telling you. If you get a 429 or a header indicating to slow down, incorporate that into your logic.

Conclusion

In this article, we explored how to implement retries for HTTP requests using Python’s requests library effectively. We began by understanding why failures happen, ranging from ephemeral network problems to server-side errors, and emphasized that intelligent retries are key (retry only when it makes sense, and avoid infinite or aggressive loops).

Looking for tips and insights or want to share your experiences with specific websites? Make sure to join our Discord community!