How to Monitor Retrieval Quality in a Production RAG System

When a retrieval-augmented generation system gives a wrong answer, the model usually gets the blame. Often the real problem happened earlier: the search returned the wrong documents, outdated ones or nothing useful at all. Monitoring retrieval separately from generation tells you where to look.

Step 1: Log what was retrieved

For each request, record the query, the IDs and scores of the retrieved chunks, the index version and which chunks the final answer cited. Store document IDs rather than full text, so logs stay small and access stays controlled.

Step 2: Track simple signals first

Some useful metrics need no grading at all:

• Weak retrievals: the share of queries where the best score falls below a threshold.
• Citation rate: how often the answer cites at least one retrieved chunk.
• Stale hits: retrieved documents older than a given age, or marked as replaced.
• Dominant documents: a few documents showing up in most results often points to a chunking or ranking problem.

Step 3: Grade relevance on a sample

Each day, take a sample of queries and ask a grading model whether each retrieved chunk is relevant, on a simple scale. Compute precision for the top results, and check the grader against human judgment on a small subset every week.

for q in daily_sample(queries, 200):
    grades = [grade_relevance(q.text, chunk.text) for chunk in q.top_chunks[:5]]
    record("retrieval_precision_at_5", sum(grades) / len(grades), route=q.route)

Step 4: Keep a fixed benchmark

Maintain a set of questions with known correct source documents. Run it after every index rebuild, embedding model change or chunking change. Recall on this set shows whether the right documents can still be found at all.

Step 5: Watch the content

Retrieval quality often drops when the underlying documents change, for example after a new policy version, a reorganized wiki or a failed sync. Alert when the document count for a source changes sharply.

Step 6: Connect to user feedback

Link thumbs-down feedback to the retrieval log for that request. Reviewing a week of negative feedback usually shows whether retrieval or generation is the bigger problem.

Things to watch

• Permissions. Graders and dashboards must respect document access rules. Don't expose restricted content in monitoring tools.
• Embedding upgrades. Changing embedding models means re-indexing everything, and scores aren't comparable across models.
• Thresholds. Similarity scores vary by model and domain, so set thresholds from your own data.

Add the weak-retrieval metric this week. It's cheap, and it often explains a surprising share of bad answers.