Lesson 11 of 12 · Calling APIs From Python
Rate Limits and Retries
Learn what API rate limits are, how to handle 429 Too Many Requests and Retry-After, and how to retry failed requests safely with exponential backoff and jitter in Python.
- 8 min read
- Beginner
- Reviewed
- What rate limits and quotas are, and why APIs have them
- How to read rate-limit headers and the Retry-After header
- Which failures are worth retrying, and which are not
- How exponential backoff and jitter work
Why APIs limit you
A popular API serves millions of requests from thousands of programs. If one buggy program sent requests in a tight loop, it could slow the service down for everyone. So APIs set limits.
AI APIs usually limit several things at once: requests per minute, tokens (pieces of text) per minute, and money per month.
Reading the limits
Many APIs tell you where you stand in the response headers:
import requests
response = requests.get(
"https://api.open-meteo.com/v1/forecast",
params={"latitude": 17.38, "longitude": 78.49, "current": "temperature_2m"},
timeout=10,
)
for name in ["X-RateLimit-Limit", "X-RateLimit-Remaining", "X-RateLimit-Window"]:
print(f"{name}: {response.headers.get(name)}")At the time of writing, Open-Meteo allows a set number of requests in each window of 60 seconds, and counts down how many you have left.
A pretend rate-limited server
This pretend server allows three requests. After that, it answers 429 and tells you to wait:
calls = 0
def limited_api():
global calls
calls += 1
if calls > 3:
return 429, {"Retry-After": "20"}, "Too many requests"
return 200, {}, f"Result {calls}"
for _ in range(5):
status, headers, body = limited_api()
if status == 429:
print("429 received. The server says wait", headers["Retry-After"], "seconds.")
else:
print(status, body)What is worth retrying?
Retrying only helps when the problem is temporary:
| Retry? | Failures |
|---|---|
| Yes | timeouts, connection errors, 429, 500, 502, 503, 504 |
| No | 400, 401, 403, 404: the same request will fail again, so fix it instead |
Remember the Methods lesson too: retrying GET, PUT and DELETE is safe, but retrying a POST might create something twice.
Exponential backoff
If a server is overloaded and every client retries instantly, the server gets even more overloaded. The polite way is to wait a little longer after each failure.
def backoff_delay(attempt, base=1, cap=30):
return min(base * 2 ** attempt, cap)
for attempt in range(7):
print(f"Attempt {attempt + 1} failed. Wait {backoff_delay(attempt)} s.")Jitter
If a thousand clients all fail at the same moment, they would all retry after exactly 1 second, then exactly 2 seconds, hitting the server together again and again.
import random
random.seed(7) # makes this demo print the same numbers every time
for attempt in range(4):
delay = 2 ** attempt
with_jitter = delay + random.uniform(0, 1)
print(f"Backoff {delay} s, with jitter {with_jitter:.2f} s") # :.2f shows 2 decimal placesA complete retry loop
Here everything comes together. The pretend server fails twice with 503, then works. The loop retries only temporary errors, respects Retry-After when it is given, and gives up after a few attempts. To keep the demo fast, wait() only prints. Real code would call time.sleep(seconds).
answers = [(503, {}), (503, {"Retry-After": "3"}), (200, {})]
def flaky_api():
# Pretend server: takes the next answer from the list.
return answers.pop(0)
def wait(seconds):
print(f" waiting {seconds} s") # real code: time.sleep(seconds)
RETRYABLE = {429, 500, 502, 503, 504}
MAX_ATTEMPTS = 5
for attempt in range(MAX_ATTEMPTS):
status, headers = flaky_api()
print(f"Attempt {attempt + 1}: status {status}")
if status < 400:
print("Success!")
break
if status not in RETRYABLE:
print("Not worth retrying. Fix the request.")
break
if "Retry-After" in headers:
wait(int(headers["Retry-After"]))
else:
wait(min(2 ** attempt, 30))
else:
print("Giving up after", MAX_ATTEMPTS, "attempts.")The else after a for loop runs only when the loop finishes without break, which here means every attempt failed.
Do not hit the limit in the first place
If you must make many requests, space them out with time.sleep(), ask for more data per request where the API allows it, and save (cache) answers you have already received instead of asking twice.
Common mistakes
Write the function that plans the waits for exponential backoff with a cap.
# Write backoff_delays(attempts, base=1, cap=20) that returns a LIST of waits:
# the wait for attempt n (starting at 0) is base * 2 ** n, but never more than cap.
delays = backoff_delays(7)
print(delays)
print("Total wait:", sum(delays), "seconds")
print(backoff_delays(4, base=3, cap=10))
# Output must be exactly:
# [1, 2, 4, 8, 16, 20, 20]
# Total wait: 71 seconds
# [3, 6, 10, 10]
Which of these failures is worth retrying after a short wait?
Summary
- A rate limit caps requests per period. A quota caps usage over a longer period. Going over gives
429. - Read the rate-limit headers, and wait as long as Retry-After says.
- Retry only temporary failures: timeouts, connection errors, 429 and 5xx. Fix 4xx errors instead.
- Use exponential backoff with a cap, plus jitter, and a maximum number of attempts.
- Throttle your own requests so you rarely hit the limit.
Next: the final project, where you build a real weather report from a public API.
Finished reading? Tick it off to track your progress.