Why Product Managers Should Set Data Freshness Targets

Your support assistant just quoted a customer the old price. It sounded sure, it cited a real document, and the document was correct in March. Retrieval worked, the model behaved, and the answer was still wrong. The fix starts with the product manager setting data freshness targets, one per source, because nobody else is placed to decide how stale is too stale.

Stale knowledge base answers are a product bug

Retrieval metrics ask whether the system found the right document, not whether that document is still true. A retriever can score perfectly while serving last quarter's price list.

In most teams, "how stale is too stale?" has no owner. Data engineering ships whatever sync schedule was easiest, and support finds out from a customer. With RAG (retrieval-augmented generation, where the model answers from documents fetched at question time), the default freshness target is "whenever someone re-indexes", so it was decided by accident.

Set data freshness targets per source, not per system

One number for the whole knowledge base is wrong for everything in it. Content goes bad at different speeds, and a wrong answer costs different amounts. Example values to show the shape:

| Source | Example max age | Why | | --- | --- | --- | | Price list | 24 hours | A wrong price becomes a contract dispute | | Return policy | 90 days | Changes rarely, but a wrong answer is expensive | | Company history page | 1 year | Nobody gets hurt |

Your numbers come from two questions: how fast does this change, and what does a wrong answer cost? Both are product questions. A freshness SLA is those answers written down per source, with a named owner. Tighter targets also mean more pipeline runs and re-embedding, so the trade-off belongs to whoever owns the budget.

Write the freshness target where the pipeline can check it

A target in a slide deck decays. A target in config gets checked. If your sources land in a warehouse, dbt's source freshness lets you declare a warning and an error threshold per table:

# models/sources.yml
sources:
  - name: knowledge_base
    database: raw
    config:
      loaded_at_field: _etl_loaded_at
    tables:
      - name: price_list
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}
      - name: return_policy
        config:
          freshness:
            warn_after: {count: 60, period: day}
            error_after: {count: 90, period: day}

This is the dbt 1.10 layout, with freshness and loaded_at_field under config:; older versions put them at the source level. Run dbt source freshness on a schedule and alert on errors.

One trap: loaded_at_field tells you when data was loaded, not when the business last changed it. A nightly job that re-copies a six-month-old price list will look perfectly fresh. Track the source's own modified date wherever you can.

Decide the fallback when data is late

The target only matters if something happens when you miss it, and that choice is the product decision. Per source, pick one: answer with a caveat, refuse and hand off to a human, or serve the last known good version with a visible banner.

A price list should refuse; a return policy can answer with a caveat. This gate runs after retrieval and before generation:

from datetime import datetime, timedelta, timezone

# Product owns this table. Engineering owns how to hit it.
FRESHNESS = {
    'price_list': {'max_age': timedelta(hours=24), 'when_late': 'refuse'},
    'return_policy': {'max_age': timedelta(days=90), 'when_late': 'caveat'},
}

def freshness_gate(chunks, now=None):
    # Each chunk: {'source': str, 'text': str, 'updated_at': aware datetime}
    now = now or datetime.now(timezone.utc)
    if not chunks:
        return {'action': 'refuse', 'message': 'I found nothing to answer from.'}

    late = [c for c in chunks
            if now - c['updated_at'] > FRESHNESS[c['source']]['max_age']]

    if any(FRESHNESS[c['source']]['when_late'] == 'refuse' for c in late):
        return {
            'action': 'refuse',
            'message': 'I cannot confirm current figures. A teammate will follow up.',
        }

    oldest = min(c['updated_at'] for c in chunks)
    return {
        'action': 'answer',
        'as_of': oldest.date().isoformat(),
        'caveat': bool(late),
    }

Feed updated_at the source's modified date, not the index date, so re-indexing old content doesn't reset its age.

Show an as-of date in every answer

Users forgive an old answer more easily when it admits its age. Pass the gate's result through to the reply:

def answer_with_as_of(question, retrieved_chunks, generate):
    gate = freshness_gate(retrieved_chunks)
    if gate['action'] == 'refuse':
        return gate['message']

    answer = generate(question, retrieved_chunks)
    as_of = gate['as_of']
    footer = f'Based on sources last updated {as_of}.'
    if gate['caveat']:
        footer += ' Some of it may be out of date.'
    return answer + '\n\n' + footer

generate is whatever function calls your model. The as-of line uses the oldest chunk, so an answer never looks fresher than its weakest source.

Isn't this a data-engineering problem?

Partly. Engineers build the checks, alerts and pipeline. But the number and the fallback are not engineering questions. Left alone, engineers pick what the pipeline can hit comfortably, which is rarely what the business needs. If a product manager asks for five-minute freshness on a nightly export, engineering will say no and put a price on it. Product owns the target and what the user sees; engineering owns how to hit it.

Your first step this week

List every source your AI feature reads. Next to each, write a max age, a fallback and a named owner. Start with the one source where a stale answer would cost real money, put its target in config, and add the as-of line to the reply.

In production, a freshness target turns "the AI said something outdated" from a surprise in a support ticket into an alert someone set up on purpose.