Why Your LLM Fallback Model Needs Its Own Evals

Your fallback is the one code path that has never served real traffic, and you're trusting it with your worst day. An LLM fallback model is supposed to catch requests when the primary provider is down or throttling. If nobody has evaluated it, you haven't built resilience. You've built a faster way to ship behavior you've never seen.

The fix is three things: run the same golden-set evals on the fallback, set a written quality floor, and alert on fallback rate.

The outage where everything returned 200 OK

Picture the 2am page. The primary provider is throwing 5xx errors, your router switches to the backup, and the dashboards go green. Error rate: zero.

Then the tickets arrive. The agent stopped calling the refund tool. Extraction jobs are writing malformed JSON. A summarizer is politely declining to summarize invoices. Nothing failed. The fallback just behaved differently.

A failover drill proves that traffic moves. It says nothing about whether the answers are still good.

What changes when you swap models

A different model is a different system, even behind the same interface:

• Tool calls. Argument names, how strictly the model follows your JSON schema, and how parallel calls are returned can all differ. A parser written against one model's output may choke on another's.
• Refusals. Safety tuning differs between models. A prompt that passes cleanly on the primary may get hedged or declined on the fallback. Or the fallback may be more permissive and skip a guardrail you were quietly relying on the model to enforce.
• Prompt fit. Your system prompt was tuned against the primary, probably over weeks. Format rules, tone and length habits don't transfer for free.
• Latency and limits. Context window, max output and time to first token vary. A smaller context window truncates your retrieval prompt. A slower model blows your timeout, triggers retries and doubles the load.
• Cold quota. The fallback takes no traffic until the day everything lands on it, so its rate limits are probably sized for nothing.

Run your golden set against the fallback

You already have a golden set for the primary. Point the same file at the fallback and score it with the same grader. This sketch assumes call_model, grade and tool_call_is_valid are helpers you already have.

import json

GOLDEN_SET = 'evals/golden.jsonl'  # the same file your primary is scored on
FLOOR = {'pass_rate': 0.85, 'tool_call_valid': 0.98}  # starting points, tune per route
MAX_P95_LATENCY_S = 8.0

def run_suite(model_id):
    passed, tool_ok, latencies = [], [], []
    with open(GOLDEN_SET) as f:
        for line in f:
            case = json.loads(line)
            out = call_model(model_id, case['messages'], tools=case.get('tools'))
            passed.append(grade(case, out))
            tool_ok.append(tool_call_is_valid(case, out))  # True when no tool call is expected
            latencies.append(out.latency_s)
    latencies.sort()
    return {
        'pass_rate': sum(passed) / len(passed),
        'tool_call_valid': sum(tool_ok) / len(tool_ok),
        'p95_latency_s': latencies[int(0.95 * (len(latencies) - 1))],
    }

scores = run_suite('<your-fallback-model-id>')
print(json.dumps(scores, indent=2))

below_floor = [k for k, v in FLOOR.items() if scores[k] < v]
if scores['p95_latency_s'] > MAX_P95_LATENCY_S:
    below_floor.append('p95_latency_s')
if below_floor:
    raise SystemExit(f'Fallback below quality floor: {below_floor}')

The floor is lower than your primary's bar on purpose, but it is written down and it fails the build. The numbers above are placeholders, so set yours from what each route can tolerate.

Run this in CI whenever prompts change, and on a schedule, because model versions get retired and updated underneath you.

The floor also forces a decision before 2am. Low-stakes routes can serve degraded answers. Routes that write to a system of record should fail closed or queue the work until the primary returns.

Alert on fallback rate, not just errors

Failover hides itself by design, so the only signal is how much traffic is on the backup. Tag every request with a route label (primary or fallback) and emit it as a counter. Then alert using a standard Prometheus alerting rule:

groups:
- name: llm-fallback
  rules:
  - alert: LLMFallbackRateHigh
    expr: |
      sum(rate(llm_requests_total{route="fallback"}[5m]))
        /
      sum(rate(llm_requests_total[5m])) > 0.05
    for: 10m
    labels:
      severity: page
    annotations:
      summary: 'More than 5% of LLM traffic is on the fallback model'
      description: 'Check the primary provider, and check that the router flipped back.'

llm_requests_total is a metric name you define, and 5% is a placeholder: pick a value above your normal baseline. The for: 10m stops a brief blip from paging anyone. This alert also catches the quieter failure, where the outage ended days ago and the router never switched back.

If you sample production outputs for scoring, split that score by route too.

The strongest objection: it's only for a few hours

Fair. For many routes, degraded beats down, and a fallback that is merely okay is a good trade. Evals also cost time and tokens.

But a golden-set run costs minutes and a modest token bill, and it is the only way to learn which routes are safe to degrade and which are not. Without an alert, a few hours can also quietly become a few days.

When it's not worth it: an internal prototype where a human reads every output anyway.

Your first step this week

Take the eval set you already run against the primary, point it at the fallback, and write down the scores. Compare them to the primary, pick a floor, and add the fallback-rate alert.

A fallback you have never scored is a guess about your worst day, and production deserves a measurement.