Why SLOs for AI Features Need More Than Uptime

Traditional service level objectives measure whether a service answers and how fast. For an AI feature, both can look perfect while users receive confident, wrong answers. If your objectives only cover uptime, the dashboard stays green during the incidents that matter most.

What standard SLOs miss

• Wrong but valid answers. The request succeeds, returns HTTP 200 and contains nonsense.
• Silent degradation. A provider model update or a retrieval change lowers quality gradually, without errors.
• Partial failures. A tool call fails, the model works around it and gives an incomplete answer that looks complete.

A better set of objectives

Keep the usual objectives and add quality:

1. Availability: 99.5 percent of requests return a response without an error.
2. Latency: 95 percent of requests receive the first token within two seconds.
3. Quality: 90 percent of sampled responses pass an automated rubric check each day.
4. Safety: no response in the sample breaks a hard rule, such as quoting a price the tools didn't return.

These numbers are examples. Agree the real ones with the product owner, based on what users can tolerate.

How to measure quality continuously

• Sample a few hundred responses a day, with personal data removed.
• Grade them against a rubric with a grading model, and check the grader against human review every week.
• Track user signals alongside, such as thumbs-down rate and escalations.

The daily pass rate becomes a service level indicator like any other, with a target and an error budget.

Why an error budget helps here too

When the quality budget is spent, the team pauses risky changes, such as new prompts or model upgrades, and works on quality instead. Without a budget, quality work tends to lose to feature work.

The strongest objection

"Automated grading isn't precise enough for an SLO." It isn't perfectly precise, and it doesn't need to be. What matters is that the measurement is consistent, so a drop from 92 to 81 percent is a real signal worth investigating. Regular human spot checks keep the grader honest.

Things to watch

• Averages hide failures. Break quality down by route or customer, because one important workflow can fail while the overall number holds steady.
• Rubric changes. Changing the rubric changes the metric. Version it and mark changes on the dashboard.
• Cost. Grading uses model calls too. Sample enough to be useful, not everything.

Pick one AI feature, write down its quality objective and measure it for two weeks before committing to targets.