How to Run a Pre-Mortem Before an AI Feature Goes Live

Your AI feature works in the demo. The pre-mortem is the meeting where you find out how it will die in production, while fixing it is still cheap. An AI feature pre-mortem is a 60-minute session where the team assumes the launch already failed, lists why, ranks the causes, and turns each one into an owner, an eval, an alert or a launch gate. You finish with a risk log and a go/no-go rule you can check in CI.

The technique comes from Gary Klein and is described in Harvard Business Review: imagine the project has failed, then explain why. It works because people hold back doubts during planning, and "it already failed" turns doubt into the assignment.

1. Pick the room for your pre-mortem meeting

Invite five to eight people: engineers, the data owner, someone from support who will receive the tickets, the security reviewer, and the customer-side owner if you work inside a client. Pick a facilitator who isn't the tech lead, and send a one-paragraph description of what ships, to whom and when.

2. Set the scene and brainstorm silently (15 minutes)

The facilitator reads this out:

It is three months after launch. The feature has been switched off and the sponsor is asking what happened. Write down why.

Everyone then writes failure reasons alone, one per note, so the loudest story doesn't anchor the room. Use AI-specific prompts:

• Data: stale, missing, duplicated or permission-blind documents.
• Model: the provider changes behavior, quality drops on real inputs, confident wrong answers.
• Access: a service account that can't see what the feature needs, or sees too much.
• Cost: usage ten times the forecast, a retry loop that never stops.
• People: nobody owns it or knows how to roll it back.

3. Cluster and rank the AI project risks (20 minutes)

Group duplicate notes for 10 minutes. Then score each group 1 to 3 for likelihood and 1 to 3 for impact, multiply, and keep the top five to seven.

Write each survivor as a one-sentence failure story, such as "the bot quotes a refund policy retired in June". Stories test better than themes like "data quality".

4. Turn each risk into a control (20 minutes)

For every risk, pick one primary control: an owner, an eval that fails when it happens, an alert someone gets paged on, a gate that must be true before launch, or a written accept with a name next to it. The result is an AI launch checklist built from your own failures.

| Failure story | Control | | --- | --- | | Bot quotes a retired policy | Eval: add retired-policy questions to the eval set. Alert on source freshness. | | Provider updates the model and answers shift | Pin the model version if the API allows. Run evals on a schedule; alert when the pass rate drops. | | Nobody owns it after handover | Gate: a named owner, a runbook and rollback steps. No name, no launch. |

5. Write the pre-mortem template into a risk log (5 minutes)

Keep the log in the repo so it gets reviewed like code. One entry per risk:

risks:
  - id: R1
    failure_story: "Support bot quotes a refund policy that was retired in June"
    category: data            # data | model | access | cost | people
    likelihood: 3             # 1 low, 3 high
    impact: 3
    control_type: eval        # owner | eval | alert | gate | accept
    control: "Add 25 retired-policy questions to the eval set; fail release under 90% pass"
    owner: "<one person, not a team>"
    due: "<yyyy-mm-dd>"
    launch_blocker: true
    status: open              # open | done | accepted

Every entry gets one owner and a due date. The launch_blocker flag marks which controls must exist before go-live.

6. Make the launch gate check itself

A gate nobody checks is a wish. This script fails a CI step while any blocker is open or unowned:

import sys
import yaml  # pip install pyyaml

with open("risk_log.yaml") as f:
    risks = yaml.safe_load(f)["risks"]

blocked = [
    r["id"]
    for r in risks
    if r.get("launch_blocker")
    and (not r.get("owner") or r.get("status") not in ("done", "accepted"))
]

if blocked:
    print("Launch blocked by:", ", ".join(blocked))
    sys.exit(1)
print("All launch blockers closed.")

Run python check_risk_log.py in your release pipeline. A non-zero exit stops the job, so the go-live conversation starts from the log, not from optimism.

Gotchas

• Don't let it turn into a design review. Park fixes until step 4, or the first problem eats the hour.
• Seniors speak last. If the sponsor goes first, the room will agree with the sponsor.
• "The team" is not an owner. One name per risk, or nothing happens.
• Score is not the only filter. Anything that leaks data or harms a customer is a launch blocker regardless of its likelihood score.
• Rerun it when the ground moves: a new model, data source or customer. The log only holds what the room could imagine, so add to it after every incident.

The failures that hurt in production are the ones everyone half-suspected and nobody owned, and an hour of imagined failure gives each of them a name and a deadline.