Benefits of Circuit Breakers for Flaky LLM API Calls

Your model provider has a bad afternoon, and your retry logic answers by tripling the traffic. Circuit breakers on LLM API calls are the cheap fix: after a few failures, stop calling for a while, serve a fallback, and probe again later. This article covers five concrete benefits, a Python sketch you can adapt, and the cases where you should skip the pattern.

How a circuit breaker works for an LLM API

Martin Fowler's description of the pattern is still the clearest. A breaker wraps a remote call and counts failures, and timeouts count as failures. It has three states:

• Closed: calls go through, failures are counted, and a success resets the count.
• Open: after enough failures, calls fail immediately without touching the provider.
• Half-open: once a reset timeout passes, one trial call goes through. Success closes the breaker, failure reopens it.

Here is a minimal version. Swap trip_on for your SDK's timeout, connection, 429 and 5xx exception classes.

import logging
import threading
import time

log = logging.getLogger("llm.breaker")


class CircuitOpenError(Exception):
    """Raised instead of calling the model while the breaker is open."""


class CircuitBreaker:
    def __init__(self, trip_on, failure_threshold=5, reset_timeout=30.0):
        self.trip_on = trip_on  # exception types that mean "the provider is struggling"
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self._failures = 0
        self._opened_at = None
        self._probing = False
        self._lock = threading.Lock()

    def call(self, fn, *args, **kwargs):
        with self._lock:
            if self._opened_at is not None:
                cooled_down = time.monotonic() - self._opened_at >= self.reset_timeout
                if not cooled_down or self._probing:
                    raise CircuitOpenError("LLM circuit open")
                self._probing = True  # half-open: let exactly one trial call through
        try:
            result = fn(*args, **kwargs)
        except self.trip_on:
            self._record(failed=True)
            raise
        except Exception:
            self._record(failed=False)  # e.g. a 400: the provider answered, so it is up
            raise
        self._record(failed=False)
        return result

    def _record(self, failed):
        with self._lock:
            self._probing = False
            if not failed:
                if self._opened_at is not None:
                    log.warning("LLM circuit closed")
                self._failures = 0
                self._opened_at = None
                return
            self._failures += 1
            if self._failures >= self.failure_threshold:
                if self._opened_at is None:
                    log.warning("LLM circuit opened after %d failures", self._failures)
                self._opened_at = time.monotonic()

This counts consecutive failures, lets one probe through after reset_timeout, and logs when it opens and closes. Put it inside your retry loop so every attempt checks the breaker first, otherwise retries keep knocking on an open circuit. Use one breaker per provider and model, not one global breaker, so a sick model doesn't blind you to a healthy one.

The call site is where the fallback lives:

breaker = CircuitBreaker(
    trip_on=(TimeoutError, ConnectionError),  # add your SDK's 429 and 5xx errors
    failure_threshold=5,
    reset_timeout=30,
)


def summarize(ticket: str) -> dict:
    try:
        text = breaker.call(call_llm, ticket, timeout=10)  # call_llm wraps your SDK
        return {"summary": text, "degraded": False}
    except (CircuitOpenError, TimeoutError, ConnectionError):
        return {"summary": ticket[:280], "degraded": True}

The degraded flag lets the UI and your logs tell a fallback from a real answer.

Benefits of circuit breakers for LLM API calls

1. Users get an answer in milliseconds, not minutes. Take a 30-second timeout and three retries. That is four attempts, so a user can stare at a spinner for two minutes before backoff delays even count. With the breaker open, the same request returns the fallback immediately, which is the point Fowler makes about not tying up resources on calls likely to fail.

2. The rest of your app keeps its capacity. Slow LLM calls hold workers, connections and threads. When they pile up, endpoints that never touch a model start timing out too, the cascade Fowler warns about. An open breaker rejects a call in the time it takes to compare two timestamps, so the number of stuck requests stops growing.

3. Retries stop making the outage worse. Providers answer overload with errors you will see in production, such as the 529 overloaded response in Anthropic's API errors reference. Backoff with jitter spreads the load out, and a breaker removes it, keeping traffic off a struggling server. It also stops you hammering limits that retrying can't lift: a monthly spend-cap 429 arrives without a retry-after header, and retries fail until access resumes.

4. The fallback path is one clean branch. A single CircuitOpenError tells you the provider is out, so degradation lives in one except clause: a cached answer, the truncated original, a rules-based reply, or a plain "try again shortly". You design the degraded experience once instead of letting every timeout handler improvise its own.

5. You get a signal worth alerting on. Fowler recommends logging every state change, alerting when the breaker trips and resets, and treating trips as early warnings of deeper problems. A status page tells you what the provider thinks. A "circuit opened" log line tells you what your own calls are experiencing, which is what your users feel.

When a circuit breaker for an LLM API is not worth it

Skip it for a low-traffic internal tool. If it handles a handful of requests an hour, five consecutive failures can take hours to accumulate, so the breaker's state is mostly stale. A sensible timeout, one or two retries and a readable error message do the job.

The same goes for overnight batch jobs that can back off and wait, and for setups where a gateway or service mesh already breaks circuits. A second breaker there is more state to debug for no gain.

User-facing AI features fail at the dependency you don't control, and a breaker is the smallest piece of code that keeps that failure from becoming yours.