Benefits of Tracking Latency Percentiles for LLM APIs

Model response times vary far more than typical API calls. A short answer might take one second, while a long one with tool calls takes thirty. The average of those numbers describes almost nobody's experience. Percentiles do.

What to measure

• Time to first token (TTFT): how long until the first part of the response appears. This decides whether a chat feels responsive.
• Total response time: how long until the full answer is ready, which matters for background tasks and APIs.
• Output speed: tokens per second once text starts arriving.

Record each as a histogram, so you can calculate the 50th, 95th and 99th percentiles.

Benefit 1: You see the slow requests people remember

If 95 percent of requests are fast but the slowest 5 percent take 40 seconds, the average can look fine while those users give up. The 95th and 99th percentiles show them directly.

Benefit 2: Streaming problems become visible

A response can finish in a reasonable total time but take ten seconds to start. Tracking time to first token separately shows whether users are staring at an empty screen.

Benefit 3: Better targets and alerts

A target like "95 percent of chat requests start streaming within two seconds" is specific and testable. Alerts on a rising percentile catch real problems without firing for one unusual request.

Benefit 4: Clearer comparisons

When you test a new model, a longer prompt or a different region, comparing percentiles shows whether the change hurt the slow end, which averages smooth over.

Benefit 5: Better conversations with providers

If you raise a latency problem with a provider, percentile data by model, region and time of day is far more useful than "it seems slow".

How to record them

With Prometheus, use a histogram with buckets suited to model calls:

from prometheus_client import Histogram

TTFT = Histogram(
    "llm_time_to_first_token_seconds",
    "Time to first token",
    ["route", "model"],
    buckets=[0.25, 0.5, 1, 2, 4, 8, 16, 32],
)

Then chart histogram_quantile(0.95, sum by (le, route) (rate(llm_time_to_first_token_seconds_bucket[5m]))) in Grafana.

When percentiles mislead

At very low traffic, a five-minute 99th percentile may rest on a handful of requests. Use longer windows, or look at the individual slow requests instead.

Add the time-to-first-token histogram first. It's the number most closely tied to how responsive your feature feels.