How to Record and Replay Production Traffic to Test AI Changes

Evaluation sets written by the team tend to reflect what the team expects users to ask. Real traffic is messier. Replaying a sample of real requests against a new version is one of the most reliable ways to see how a change behaves before it ships.

Step 1: Confirm what you may record

Check your privacy policy and customer contracts before storing any requests. Many enterprise agreements restrict how conversation data can be used. Get agreement in writing and document how long recordings are kept.

Step 2: Capture requests with their context

Record the input and everything needed to reproduce the call:

• The user message and conversation history.
• The prompt version and model settings.
• Tool results returned during the conversation.
• The response and any user feedback.

Tool results matter most. Replayed without them, the new version calls live tools and you end up testing different data.

Step 3: Remove sensitive data

Run recordings through a redaction step before storing them. Replace names, email addresses, account numbers and other identifiers with placeholders. Test the redaction on real samples, because pattern matching misses more than people expect.

Step 4: Replay against the candidate

for rec in sample(recordings, 300):
    tools = FakeTools(rec.tool_results)
    new = candidate.run(rec.messages, tools=tools)
    store_comparison(rec.id, old=rec.response, new=new)

Serving the stored tool results through fake tools means both versions see identical data.

Step 5: Compare the results

Grade the pairs against a rubric, with a model grader for scale and human review for a sample. Look closely at the cases where the new version is worse, not only at the average score.

Step 6: Keep the sample fresh

Take a new sample every few weeks. Usage shifts, and recordings from six months ago no longer represent your users.

Things to watch

• Replays cost money. A few hundred well-chosen requests usually tell you enough.
• Multi-turn conversations diverge after the first changed reply. Compare single turns, or stop at the first turn that differs.
• Keep recordings in a restricted store with access logging.

Start with 100 redacted single-turn requests and replay them before your next prompt release.