Benefits of Architecture Decision Records for AI Projects

Six months after you roll off the engagement, someone on the customer's team asks why the assistant runs on that model, in that region, with that chunk size. Nobody remembers, the Slack thread is long gone, and a simple question becomes a two-week investigation. Architecture decision records for AI projects fix that: one short markdown file per decision, written when you make it, readable by people who weren't in the room.

An architecture decision record (ADR) captures one significant decision with its context, the options you ruled out and the consequences. Microsoft's Azure Well-Architected guidance says to record only choices that affect the system's structure, affect key quality attributes, or are hard to reverse. AWS Prescriptive Guidance calls the collection of ADRs the decision log, and the ADR community site hosts example templates.

An architecture decision record template for AI projects

On an AI project, the decisions worth a file are the ones the customer will live with: which model, which region, build versus buy, retrieval design (chunking, vector store, reranking), guardrails, and the eval gate that decides when a change ships. Status, options considered and confidence follow Microsoft's suggested fields. Revisit when is my addition, and benefit 5 shows why.

# ADR-<number>: <the decision as a short statement>

Status: Proposed | Accepted | Superseded by ADR-<number>
Date: <yyyy-mm-dd>
Owner: <name>
Stakeholders: <customer security, data owner, ...>

## Context
<The problem, plus the constraints that forced our hand: data residency,
latency budget, price ceiling, team skills, contract terms.>

## Options considered
1. <Option A>: <why not>
2. <Option B>: <why not>
3. <Chosen option>: <why>

## Decision
We will <do X>.

## Consequences
- Good: <...>
- Costs and risks: <...>
- Confidence: low | medium | high, because <evidence, e.g. link to eval run>

## Revisit when
<Trigger: model retirement, price change, new regional launch, eval score
below the agreed threshold.>

Keep each record to about a page and link to the eval report instead of pasting it in. A filled-in example, with illustrative content:

# ADR-004: Run the claims assistant on a hosted model API in an EU region

Status: Accepted
Date: <yyyy-mm-dd>
Owner: <name>

## Context
Claim documents contain personal data. The customer's security policy
requires processing inside the EU. The pilot used a US endpoint.

## Options considered
1. Self-host an open-weight model: the customer has no GPU capacity or on-call rota.
2. Keep the US endpoint: fails the residency requirement.
3. Hosted API pinned to an EU region: meets residency, no GPUs to run.

## Decision
We will call a hosted model API pinned to an EU region.

## Consequences
- Good: no inference infrastructure to hand over.
- Costs and risks: model choice is limited to what the provider offers in that region.
- Confidence: medium. Passed the acceptance set, but only on one model.

## Revisit when
The provider retires the model, or the customer builds GPU capacity.

Five benefits of keeping a decision log

1. Faster onboarding

AWS describes the decision log as a two-speed read: project members skim the headlines of each ADR for the project context, then read the ones they need in depth. A new engineer, or the customer's new hire, can skim a dozen titles in one sitting and know the shape of the system. Code shows what you built. It never says why the retrieval store is the one it is.

2. Fewer re-litigated decisions

AI projects get the same questions on a loop: why not a bigger model, why not fine-tune, why not the vector store we already run? Microsoft warns that a decision that is never recorded is likely to be forgotten, which leads to repeated debates. AWS adds that recording the reason helps stop people who weren't involved from overruling a decision later, and it asks owners to write a reason on rejected ADRs for the same purpose. Put the evidence for each ruled-out option in the options list with a link to the eval run, and the answer to "why not X" is a URL.

3. A clean audit trail

Both sets of guidance treat accepted records as fixed. If a decision changes, you write a new record and mark the old one superseded, so the history survives. When a security reviewer asks when the team agreed to send customer documents to a third-party API, and who signed off, the answer is an Accepted record. AWS has the owner add a timestamp, a version and a list of stakeholders at acceptance, and Git history supplies the rest.

4. Easier handover

A shadow period shows the customer's team how the system runs. ADRs tell them why it is built that way. Store the files in the repo next to the code and link the folder from the main docs page: AWS recommends a central location, with Git for versioning or a wiki for browsing. AWS also describes ADRs in code review, where a reviewer who spots a violating change links the ADR. After handover, that lets the customer's own engineers defend the design without calling you.

# list the decisions still in force
grep -l '^Status: Accepted' docs/adr/*.md

Records marked Superseded or Rejected drop out of that list, which is the point.

5. Cheaper model migrations

Models get retired, repriced and replaced. The hard part of moving is rarely the API change. It is knowing what the replacement must satisfy, and the Context section already lists that: residency, latency budget, price ceiling. The ADR becomes the candidate checklist and the acceptance test. The confidence line points at the weakest assumption first (ADR-004 above was tested on one model), and Revisit when tells you when to look.

The migration itself is a new record:

# ADR-012: Move the claims assistant to <new-model>

Status: Accepted
Supersedes: ADR-004

Then change only the status line of ADR-004 to Status: Superseded by ADR-012. The old reasoning stays in the log, so two years on nobody wonders why the first model was picked.

When architecture decision records are not worth it

Skip them for a throwaway prototype: a spike you will delete next week, a demo nobody inherits, any choice you can reverse in an afternoon. A few lines in the README are enough. The moment someone asks that prototype to go to production, write the hard-to-reverse records first: data residency, vendor and retrieval store.

Budget for the overhead too. AWS says the process is intense at the start of a greenfield project and recommends regular review meetings, with ADRs stabilizing in two or three sprints. It also won't fix non-compliant legacy code.

An AI system's model, region and retrieval design will all change eventually, and a decision log is how the next engineer changes them on purpose instead of by accident.