Why Retrieval Quality Belongs in Your Sprint Metrics

Your team ships a RAG feature, the demo goes great, and three sprints later support tickets start piling up because the chatbot confidently answers questions using documents from the wrong product line. Nobody touched the retriever. Someone reindexed the knowledge base, chunk sizes drifted, or a new document type got dumped into the same vector store without anyone checking how it split. Velocity stayed green the whole time.

That's the problem with treating retrieval quality as a production-only concern. By the time a monitoring dashboard flags it, you're debugging in an incident channel instead of catching it in code review. Retrieval quality sprint metrics fix that by turning retrieval into something the team plans against, the same way it plans against test coverage or cycle time.

Velocity hides retrieval regressions

Story points measure whether work got done, not whether the system still answers correctly. A ticket to "add PDF support to the ingestion pipeline" can close as done, pass code review, and still quietly drop your retrieval precision from 0.85 to 0.6 because the new PDF chunker splits tables into garbage fragments. Nothing in a standard agile board catches that. The story is closed. The regression ships anyway.

Compare that to how teams already treat test coverage or performance budgets: nobody lets a sprint end with coverage silently dropping 20 points, because it's a tracked number on the board. Retrieval quality deserves the same treatment, because a RAG system with a broken retriever is a lot like an app with a broken test suite. Everything looks fine until a real user hits the gap.

What to actually track at sprint level

You don't need a fully staffed ML eval team to do this. Pick two or three retrieval metrics that map to what your system actually promises:

• Precision@k: of the top k chunks returned, how many are actually relevant to the query.
• Recall@k: of the relevant chunks that exist in your corpus, how many made it into the top k.
• Faithfulness or groundedness: whether the generated answer is actually supported by the retrieved chunks, not hallucinated on top of them.

Open-source eval frameworks like RAGAS compute these against a small, versioned golden set of query/expected-passage pairs, which is the part that matters most: the golden set is a fixture your team owns and updates, not a live production sample.

# sprint-retrieval-eval.yaml — run in CI on every PR that touches
# the ingestion pipeline, chunking config, or embedding model
name: retrieval-quality-gate
on:
  pull_request:
    paths:
      - "ingestion/**"
      - "retriever/**"
      - "eval/golden_set.jsonl"
steps:
  - run: ragas-eval --dataset eval/golden_set.jsonl \
      --retriever-config retriever/config.yaml \
      --metrics precision_at_k,recall_at_k,faithfulness \
      --fail-below precision_at_k=0.75,recall_at_k=0.70

That gate turns a retrieval regression into a failed CI check instead of a support ticket. In planning, the number from the last run becomes a line on the sprint report right next to velocity and bug count, so "we shipped the PDF ingestion story" and "precision@5 dropped from 0.85 to 0.6" show up in the same retro.

The honest objection: isn't this just production monitoring?

No, and the distinction matters. Production monitoring tells you the retriever is degrading after real users hit it, often days after the change that caused it shipped. A sprint-level metric, backed by a golden set that runs in CI, catches the regression at the pull request that introduced it, before it's anyone's incident. You still want production monitoring for drift you can't predict, like a new class of user query nobody wrote a test for. But planning input and production alerting are two different jobs, and a lot of teams only build the second one.

There's a real cost objection too: building and maintaining a golden set takes real effort, and small teams will feel that. The fix isn't to skip it, it's to start small. Twenty query/passage pairs pulled from real support tickets beats zero, and you grow the set every time a retrieval bug makes it to production.

Start next sprint

Pick ten real user queries, pair each with the passage that should answer it, and run precision@5 against your current retriever. Put that one number on the board next to your velocity chart. If it's a start for a broader system, it's usually worth backing it with an incident before you decide what "good" looks like.

A RAG system that answers confidently and wrongly doesn't fail loud. It fails quiet, sprint after sprint, until someone escalates. Tracking retrieval quality where planning happens is how you catch it while it's still a code review comment instead of an incident.