Why LLM Retry Storms Need Jitter and a Retry Budget

LLM retry storms start small. A model API returns a few 429s, every client retries on the same schedule, and a ten-second slowdown becomes an outage you helped build. Four habits stop it: jittered backoff, honoring Retry-After, a shared retry budget, and retrying in one layer only.

Why LLM retry storms hurt more than ordinary ones

A retry against a model holds a worker for seconds and re-sends the whole prompt, so retries pile up fast while new requests queue.

Anthropic enforces limits over short intervals: 60 requests per minute can be enforced as one request per second, so a synchronized burst of retries trips the limit even when average traffic is fine. And a 529 overloaded_error can mean the API is busy across all users, so your retries join everyone else's.

In Google's SRE book, a backend capped at 10,000 QPS receives 10,100 from its frontend, and the rejected requests come back as retries: 100 QPS of retries becomes 200, then 300. A 1% overload feeds itself.

Add jitter to exponential backoff

Plain exponential backoff spaces retries out, but 500 clients that failed together all wait 1 second, then all wait 2, and hit the API in synchronized waves. Jitter randomizes the wait so the waves smear out. In AWS's simulation of 100 contending clients, jitter cut the call count by more than half.

Full jitter sleeps a random time between zero and the exponential ceiling: random.uniform(0, min(cap, base * 2 ** attempt)).

Honor the Retry-After header, then add jitter on top

When a 429 carries retry-after, it is the number of seconds to wait, and the docs say earlier retries will fail. Treat it as a floor and add a small random pad so clients that got the same value don't return together.

Two traps. Not every 429 carries the header, so keep your backoff as the fallback. And the monthly spend-cap 429 never carries it and keeps failing until access resumes, so check for enforced_spend_limit_reached and don't retry that one.

Give the whole service a retry budget

Backoff limits how fast one request retries, not how many requests retry at once. A retry budget limits that.

The SRE book suggests a server-wide budget, for example 60 retries per minute in a process. A token bucket does the same job: each retry spends a token, each success returns a fraction of one. In healthy traffic the bucket stays full; in a real incident it drains and your service fails fast instead of adding load.

Retry in exactly one layer

Retries multiply across layers. The SRE book's example: three layers that each retry 3 times mean 4^3 = 64 attempts on the database for one user action.

The Anthropic SDK retries twice by default and honors retry-after. Wrap it in your own loop of 5 attempts and one logical call can send 15 HTTP requests (5 x 3). Keep retries at the layer closest to the model call, since it sees the status codes and headers, and switch the others off.

A retry wrapper with jitter, Retry-After and a budget

import random
import threading
import time

import anthropic

# The SDK retries 429s and 5xx twice by default. Turn that off so this
# wrapper is the only retry layer.
client = anthropic.Anthropic(max_retries=0)

RETRYABLE = {429, 500, 502, 503, 504, 529}


class RetryBudget:
    """Token bucket shared by every call in this process."""

    def __init__(self, capacity: float = 50, refill_per_success: float = 0.1):
        self.capacity = capacity
        self.refill = refill_per_success
        self.tokens = capacity
        self._lock = threading.Lock()

    def try_spend(self) -> bool:
        with self._lock:
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

    def on_success(self) -> None:
        with self._lock:
            self.tokens = min(self.capacity, self.tokens + self.refill)


budget = RetryBudget()


def call_llm(max_attempts: int = 4, base: float = 1.0, cap: float = 30.0, **kwargs):
    for attempt in range(max_attempts):
        try:
            message = client.messages.create(**kwargs)
            budget.on_success()
            return message
        except anthropic.APIConnectionError as e:
            err, retry_after = e, None
        except anthropic.APIStatusError as e:
            if e.status_code not in RETRYABLE:
                raise
            if "enforced_spend_limit_reached" in e.response.text:
                raise  # monthly spend cap: retrying cannot help
            err, retry_after = e, e.response.headers.get("retry-after")

        if attempt == max_attempts - 1 or not budget.try_spend():
            raise err  # out of attempts or out of budget: fail fast

        if retry_after is not None:
            time.sleep(float(retry_after) + random.uniform(0, 2))
        else:
            time.sleep(random.uniform(0, min(cap, base * 2**attempt)))

Call it like client.messages.create, for example call_llm(model="claude-sonnet-5-5", max_tokens=1024, messages=[...]). The bucket lives in one process, so 40 replicas can still spend 40 times the budget; size it with that in mind.

The objection: a budget fails requests that would have worked

True, and that is the price: some retries that would have succeeded get dropped.

But an empty bucket means failures are widespread, so each retry has worse odds and takes capacity from calls that would have worked. Failing fast lets callers fall back, queue or tell the user, instead of waiting through a retry chain that ends in the same error.

First step: count your retry layers

Search the SDK client config, HTTP middleware, queue workers and gateway for retry settings. In one service, set max_retries=0, move retries into a single wrapper, and graph retries per successful call. Flat and near zero is healthy, and a spike is your early warning.

Unmetered retries turn the provider's bad minute into your bad hour, and rationing them is cheap insurance for any AI service in production.