Why Product Managers Should Set Claude Latency Budgets

Your spec says "summarize the ticket." It doesn't say whether the user waits one second or fifteen. Engineers fill that gap with a guess, and the guess usually becomes the product. A Claude latency budget is a number you write into the requirements before anyone picks a model.

The pain: nobody owns the wait

A support agent clicks a button and stares at a spinner. Someone files a bug that says "AI feature is slow." Engineering says it's the model. Product says it was never specified. Both are right.

Latency is part of the experience, like layout or copy. If it isn't in the spec, you'll find out about it from users.

What a latency and cost budget looks like

One short block per feature is enough. It states what the user feels and what you'll pay for it.

feature: ticket-summary
user_wait:
  first_visible_text_p95_ms: 1500   # time to first token, streamed
  complete_p95_ms: 6000
output:
  max_tokens: 250
cost:
  max_per_request_usd: <your-number>
model_tier: <chosen-after-testing>
fallback: show_cached_summary

Note the two timing numbers. With streaming, users judge the wait by when text starts, not when it ends. A summary that starts in a second and finishes in five feels fast. The same summary arriving all at once after five seconds feels broken.

Choose the Claude model tier against the budget

Anthropic sells tiers that trade capability for speed and price. The models overview lists the current ones with their latency and pricing comparisons. In general, a smaller model such as Haiku 4.5 answers faster and costs less, and a larger one such as Sonnet 5.5 reasons better and costs more.

Without a budget, the debate is taste: "Sonnet feels smarter." With one, it's a test. Run your real prompts on the cheapest tier first. If quality passes your eval, you're done. If it fails, move up a tier and check whether the budget still holds. If it doesn't, you've found a real product decision: loosen the budget, cut scope, or change the UX.

Settings that move the numbers

Two settings do most of the work, and both are product-visible:

• max_tokens. Output length is the biggest driver of total time, because the model generates text token by token. A 250-token cap is a UX decision as much as a technical one.
• Streaming. It doesn't make the model faster, but it makes the wait feel shorter.

Your engineer can measure both in a few lines:

import time
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
start = time.perf_counter()
first_token_s = None

with client.messages.stream(
    model="claude-haiku-4-5-20251001",
    max_tokens=250,
    messages=[{"role": "user", "content": "Summarize this ticket in two sentences: <ticket-text>"}],
) as stream:
    for _ in stream.text_stream:
        if first_token_s is None:
            first_token_s = time.perf_counter() - start
    final = stream.get_final_message()

total_s = time.perf_counter() - start
print(f"first token: {first_token_s:.2f}s, total: {total_s:.2f}s, output tokens: {final.usage.output_tokens}")

This logs time to first token, total time and output tokens per call. Run it on a few dozen real inputs and compare the p95 against your budget, not the average. Users remember the slow ones.

The objection: latency is an engineering detail

The strongest version: product owns the what, engineering owns the how, and seconds are a how.

But users don't experience "how." They experience a button that's either snappy or sluggish, and the choice between a fast tier and a smart tier changes what the feature can do. That is a product trade-off, so someone accountable for the product has to make it. You don't need to pick the model. You need to set the number the model is picked against, and engineering stays free to hit it any way it can.

Your first step

Pick one AI feature already in your backlog. Write the budget block above for it: first-text time, total time, max output, cost per request. Hand it to engineering and ask them to report measured p95 against it.

A latency budget turns "make it fast" into something you can test before launch, which is the difference between a pilot that survives production and one that gets switched off.