Why AI Pipelines Need Versioned Data as Well as Versioned Code

Engineering teams are used to versioning code: every change is recorded, reviewed and reversible. In AI systems, though, behavior depends just as much on data, including training sets, fine-tuning examples, evaluation sets and the documents behind retrieval. When that data changes without a record, results become impossible to reproduce.

What goes wrong without it

• Regressions nobody can explain. Accuracy drops after retraining, and nobody knows whether the code, the labels or the source documents changed.
• Evaluations that quietly change. Someone edits the evaluation set, scores improve, and the improvement turns out to be an easier test.
• Audit questions without answers. A regulator or customer asks which data trained the model that was in production last March.
• Experiments you can't repeat. A promising result from a month ago can't be recreated because the dataset was overwritten.

What versioning data looks like

Large datasets don't belong in Git itself, but references to them can. Tools such as DVC keep small metadata files in Git that point at data in object storage:

dvc add data/finetune-examples.jsonl
git add data/finetune-examples.jsonl.dvc .gitignore
git commit -m "Add 2,400 reviewed fine-tuning examples"
dvc push

Checking out an older commit and running dvc pull restores the data exactly as it was. Table formats such as Delta Lake and Apache Iceberg offer similar time travel for tabular data, and object storage versioning covers simpler cases.

Connect data versions to results

Log the data version with every training run, evaluation run and index build, alongside the code commit and model version. A model registry entry should answer "which data produced this?" without detective work.

The strongest objection

"Our data changes constantly, so versioning it is impractical." Not every change needs a new version. Snapshot the data at the points that matter: before training, before an index rebuild and whenever an evaluation set changes. That's a handful of versions a month, not thousands.

Things to watch

• Personal data. Old versions of data containing personal information are still covered by deletion requests. Plan how to remove records from them.
• Storage costs. Keep old versions as long as you need them for reproducibility or audits, then expire them.
• Evaluation sets. Review changes to them as carefully as code changes.

Start by versioning your evaluation set. It's small, it changes rarely, and it's the data whose silent changes mislead teams the most.