Python Requests With Retry#
There're several solutions to retry a HTTP request with Requests module, some of them are:
- Native Requests' retry based on urllib3's HTTPAdapter.
- Third party module: backoff.
- Third party module: tenacity.
The native HTTPAdapter is not easy to use. The tenacity module is very powerful, but is also more or less overkill because it's a general Python retry utility, and doesn't throw the same exception requests.exceptions.HTTPError
raised by raise_for_status()
of Requests. Using tenacity to an ongoing project might involve some code refactoring. So this post will just show some snippets to make retry with the backoff module.
Warning
Usually, we should only retry on idempotent verbs, we can get the same thing twice but we don't want to create the same thing twice. On the other hand, sometimes the specific environment that we're working on might have a POST as idempotent too, so make sure of that before using the retry.
Using backoff to retry#
import logging
from logging import Logger
import backoff
import requests
from requests.exceptions import HTTPError
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# in an internal enterprise environment, we often need to disable the proxy and ignore the ssl check. Of course, if you don't trust the target, then verify the ssl.
NO_PROXY = {"http": None, "https": None}
COMMON_REQUESTS_PARAMS = {"verify": False, "proxies": NO_PROXY}
# This snippet only retries on the response return code >= 500
def fatal_code(e):
return 400 <= e.response.status_code < 500
BACKOFF_RETRY_ON_EXCEPTION_PARAMS = {
# expo: [1, 2, 4, 8, etc.] https://github.com/litl/backoff/blob/master/backoff/_wait_gen.py#L6
"wait_gen": backoff.expo,
# HTTPError raised by raise_for_status()
# HTTPError code list: https://github.com/psf/requests/blob/master/requests/models.py#L943
"exception": (HTTPError,),
"max_tries": 4,
"max_time": 50, # nginx closes a session at 60' second by default
"giveup": fatal_code,
}
@backoff.on_exception(**BACKOFF_RETRY_ON_EXCEPTION_PARAMS)
def request_with_retry(
should_log: bool = False,
logger: Logger = logging.getLogger(),
logger_level: str = "info",
**request_params
):
full_params = COMMON_REQUESTS_PARAMS | request_params
requests_params_keys_to_log = ["data", "json", "params"]
if should_log:
params_message = ""
for key in requests_params_keys_to_log:
if key in request_params:
params_message += " with {} {}".format(key, request_params[key])
log_message = "[{}] {} with params{}.".format(
full_params["method"], full_params["url"], params_message
)
getattr(logger, logger_level.lower())(log_message)
response = requests.request(**full_params)
response.raise_for_status()
return response
# how to use:
request_params = {"method": "get", "url": "http://localhost"}
response = request_with_retry(**request_params)