Python Requests With Retry

Created: January 22, 2021  |  1 minute read

There’re several solutions to retry a HTTP request with Requests module, some of them are:

  1. Native Requests’ retry based on urllib3’s HTTPAdapter.
  2. Third party module: backoff.
  3. Third party module: tenacity.

The native HTTPAdapter is not easy to use. The tenacity module is very powerful, but is also more or less overkill because it’s a general Python retry utility, and doesn’t throw the same exception requests.exceptions.HTTPError raised by raise_for_status() of Requests. Using tenacity to an ongoing project might involve some code refactoring. So this post will just show some snippets to make retry with the backoff module.

Usually, we should only retry on idempotent verbs, we can get the same thing twice but we don’t want to create the same thing twice. On the other hand, sometimes the specific environment that we’re working on might have a POST as idempotent too, so make sure of that before using the retry.

Using backoff to retry

import logging
from logging import Logger

import backoff
import requests
from requests.exceptions import HTTPError
import urllib3


# in an internal enterprise environment, we often need to disable the proxy and ignore the ssl check. Of course, if you don't trust the target, then verify the ssl.
NO_PROXY = {"http": None, "https": None}
COMMON_REQUESTS_PARAMS = {"verify": False, "proxies": NO_PROXY}

# This snippet only retries on the response return code >= 500
def fatal_code(e):
    return 400 <= e.response.status_code < 500

    # expo: [1, 2, 4, 8, etc.]
    "wait_gen": backoff.expo,
    # HTTPError raised by raise_for_status()
    # HTTPError code list:
    "exception": (HTTPError,),
    "max_tries": 4,
    "max_time": 50,  # nginx closes a session at 60' second by default
    "giveup": fatal_code,

def request_with_retry(
    should_log: bool = False,
    logger: Logger = logging.getLogger(),
    logger_level: str = "info",
    full_params = COMMON_REQUESTS_PARAMS | request_params
    requests_params_keys_to_log = ["data", "json", "params"]
    if should_log:
        params_message = ""
        for key in requests_params_keys_to_log:
            if "key" in request_params:
                params_message += " with {} {}".format(key, request_params[key])
        log_message = "[{}] {} with params{}.".format(
            full_params["method"], full_params["url"], params_message
        getattr(logger, logger_level.lower())(log_message)
    response = requests.request(**full_params)
    return response

# how to use:
request_params = {"method": "get", "url": "http://localhost"}
response = request_with_retry(**request_params)

Leave a comment