Why Software Teams Need Data Contracts for AI Pipelines

A backend engineer renames user_status to account_status in a table nobody thinks of as "AI infrastructure." The migration ships clean, tests pass, the sprint moves on. Three weeks later your retrieval pipeline is confidently answering questions with stale, null-filled context, and nobody connects the dots because the failure shows up as a vague drop in answer quality, not an error.

This is the story behind most AI pipeline incidents that don't involve a model at all. The model is fine. The data feeding it quietly changed shape, and nothing was watching.

The gap that AI pipelines expose

Traditional software has a forgiving relationship with schema change. Drop a column and a query throws, a test fails, someone gets paged. The feedback loop is fast and loud.

AI pipelines don't work that way. A RAG system reading from a product catalog table doesn't crash when a field goes from a string to a nested object. It embeds whatever it gets. A null where a description used to be still produces a vector. A truncated field still gets chunked and indexed. The retrieval step returns something for every query, so there's no error to catch, just a slow erosion of relevance that shows up in user complaints, not in your monitoring.

The root problem is ownership. A table's schema is usually treated as an implementation detail of whoever writes to it. The team that owns the orders table doesn't know your embedding job reads it nightly. Nobody told them, and nothing enforces it.

What a data contract actually is

A data contract is an explicit, versioned agreement between the producer of a dataset and its consumers: field names, types, nullability, semantic meaning, and update cadence, checked automatically before a change ships.

It's the schema equivalent of an API contract. You wouldn't let a backend team change a public API's response shape without a version bump and a deprecation window. A database table feeding a production AI system deserves the same discipline as a REST endpoint.

A minimal contract for a table feeding embeddings might look like this:

# contracts/product_catalog.yaml
name: product_catalog
owner: catalog-team
version: 2.1.0
consumers:
  - team: search-platform
    pipeline: product-embeddings-nightly
schema:
  - name: product_id
    type: string
    nullable: false
  - name: description
    type: string
    nullable: false
    semantic: "used verbatim for embedding text"
  - name: status
    type: string
    nullable: false
    allowed_values: ["active", "discontinued", "draft"]
change_policy:
  breaking_change_notice_days: 14
  requires_consumer_signoff: true

The semantic field matters as much as the type. A type checker will tell you description is still a string after a change. It won't tell you the team quietly switched it from plain text to HTML, exactly the kind of change that degrades embedding quality without breaking anything downstream.

Enforce it, don't just document it

A contract that lives in a wiki page is a contract nobody reads until after the incident. It needs to run in CI, on the producer's side, before the change merges.

# validate_contract.py
import yaml
from great_expectations.dataset import PandasDataset

def validate_against_contract(df, contract_path):
    with open(contract_path) as f:
        contract = yaml.safe_load(f)

    ds = PandasDataset(df)
    for field in contract["schema"]:
        col = field["name"]
        if not field["nullable"]:
            result = ds.expect_column_values_to_not_be_null(col)
            if not result.success:
                raise ValueError(
                    f"Contract violation: {col} has nulls, "
                    f"breaks consumers: {contract['consumers']}"
                )
        if "allowed_values" in field:
            ds.expect_column_values_to_be_in_set(col, field["allowed_values"])

Run this in the producer team's pipeline, against the table they're about to write, before it lands. The specific tool doesn't matter: Great Expectations works, so does a hand-rolled check, so does a schema registry with compatibility rules if you're on Kafka. What matters is that the check lives upstream of the AI pipeline, so a breaking change gets caught by the team causing it, not discovered by the team consuming it.

The honest objection: isn't this just versioned data?

If your reaction is "we already track schema changes with versioned tables and migrations," you're half right. Versioning tells you what changed and lets you roll back. A contract tells you who it breaks before it ships, and requires their signoff to proceed. Versioning is a record. A contract is a gate.

You can have pristine version history and still ship a breaking change straight through it, because nothing in a version log stops the migration from merging. The gate is the part that actually prevents the incident.

Where this isn't worth the overhead

Don't write a formal contract for a table one person queries for an internal dashboard. The overhead of change_policy blocks and signoff requirements is worth it exactly where a schema change can silently degrade a production AI system with no error thrown: anything feeding embeddings, retrieval, feature stores, or fine-tuning data. Start there, not everywhere.

Start with your riskiest table

Pick the one upstream table that, if it silently changed shape tomorrow, would take the longest for anyone to notice. Write a contract for that one first, wire it into the producer's CI, and get the consumer team's name on it. Repeat for the next table once the first one holds.

Get this wrong and you won't get an outage. You'll get a slow, undiagnosed decline in the thing your AI system exists to do, which is worse, because nobody will think to blame the data.