Why Every AI Feature Needs a Golden Test Set

Someone tweaks a prompt to fix one annoying answer. Three days later a customer reports that a different answer, one nobody looked at, has gone sideways. If that sounds familiar, you need a golden test set.

A golden test set is a small, hand-picked collection of inputs paired with what a good output must (and must not) contain. You run it on every prompt edit, model swap or retrieval change. It is the cheapest form of LLM regression testing there is.

The pain: you find out from users

LLM behavior is a coupled system. A prompt change, a model version bump, a new chunking size or a tool description edit can each shift outputs in places you weren't looking. Eyeballing three examples in a playground tells you about those three examples.

Without a fixed set of cases, every change is judged by whatever you happened to try last. Your users become the test suite, and they file bugs slowly and angrily.

What a golden test set looks like

Keep it small: 30 to 100 cases is plenty to start. Each case is a real input plus cheap, checkable expectations. Pull them from production logs, support tickets and the bugs you have already fixed.

A good mix:

• Happy paths the feature must never fumble.
• Past incidents, so the same bug can't return quietly.
• Edge cases: empty input, wrong language, a hostile prompt.
• Refusals: things the feature must decline to do.

Store it as plain JSONL next to your code, so changes show up in code review:

{"id": "refund-window", "input": "Can I return an item after 45 days?", "must_include": ["30 days"], "must_not_include": ["yes, you can"]}

Each line is one case. The id makes failures easy to find, and the string checks are deliberately dumb because dumb checks are stable and fast.

Run the golden test set on every change

The runner can be tiny. This one assumes your feature is a function that takes a string and returns a string:

import json
import sys
from pathlib import Path

from my_app import answer  # your feature: str -> str

THRESHOLD = 0.90


def passes(case, output):
    text = output.lower()
    has_required = all(s.lower() in text for s in case.get('must_include', []))
    has_banned = any(s.lower() in text for s in case.get('must_not_include', []))
    return has_required and not has_banned


lines = Path('golden.jsonl').read_text().splitlines()
cases = [json.loads(line) for line in lines if line.strip()]
failures = [c['id'] for c in cases if not passes(c, answer(c['input']))]

score = 1 - len(failures) / len(cases)
print(f'{score:.0%} passed. Failing: {failures}')
sys.exit(0 if score >= THRESHOLD else 1)

The non-zero exit code is the point: wire python run_golden.py into CI and a regression blocks the merge. Set the threshold below 100% at first, since model output varies between runs, then tighten it as you learn which cases are flaky. For fuzzier qualities such as tone or faithfulness, you can add a model-graded check later. Start with the string checks anyway.

The objection: curated sets go stale

This is the strongest argument against, and it is partly right. A frozen set drifts away from what users actually ask, and a feature can ace the golden set while real traffic suffers.

The answer is to treat the set as a living file, not a monument. Make one rule: every production bug becomes a new case before the fix ships. Review the set monthly, retire cases for features you removed, and sample fresh inputs from logs (with personal data scrubbed) to replace the ones that no longer reflect reality.

A slightly stale set that runs on every change beats a perfect set that never runs. The golden set also doesn't replace production monitoring. It catches the regressions you can predict, and monitoring catches the rest.

Gotchas

• Flaky cases. If a case fails one run in five, fix the check or lower the temperature for tests, don't ignore it.
• Leaking into prompts. Don't paste golden inputs into your prompt as examples, or you are grading the feature on its own answer key.
• Cost. 100 cases per change is cheap, but a full-model run on every commit adds up. Run a smaller subset on commits and the full set before merge.
• Averages hide disasters. A 95% score can contain the one case your biggest customer cares about. Mark a few cases as must-pass.

Your first step this week

Open your support tickets, pick the 20 worst AI failures from the last quarter and turn each into a JSONL line. Run them against today's prompt, and you have a baseline you didn't have yesterday.

If you deploy AI to production, the golden set is what turns "I think this prompt is better" into a claim you can check before your users do.