Why Every Launch Plan Needs a Game Day Rehearsal

Your launch plan has a test column, and every row is green. Picture the first week after go-live: an API key expires at 2 a.m., the alert lands in a channel nobody watches, and the first person to notice is the customer.

A game day rehearsal is how you find that out in advance. You break something on purpose in a production-like environment, with the real team on the call, and time how long they take to notice and recover. Below you get the case for it, a one-page plan and the command that starts your first rehearsal.

Your go-live checklist tests the system, not the response

Tests answer "does it work?" Launch readiness asks a different question: when it stops working, will we know, and will we know what to do? A checklist can record that a runbook exists. It can't record whether anyone can follow it at 2 a.m.

AWS's Well-Architected guidance on game days lists "you document your procedures, but you never exercise them" as an anti-pattern. It recommends involving the same teams who would handle the real event. Google's SRE book describes a similar ritual, the Wheel of Misfortune, where on-call engineers work through a made-up outage.

Rehearsals find the boring gaps. The alert goes to the wrong channel. The only person with admin rights is on leave. The rollback needs a permission nobody on the call holds. The customer's identity-provider contact is missing from the escalation list. None of these show up as a failed test.

Add a game day rehearsal to your go-live checklist

A game day is a scheduled, time-boxed session. A facilitator injects a failure, and responders handle it as if it were real. As the project manager you don't run the fault. You put the rehearsal on the plan as a gate with an owner and a date, the same way you treat security sign-off.

game_day:
  name: Pre-launch rehearsal for <system-name>
  date: <yyyy-mm-dd>
  length: 90 minutes
  environment: <staging, with the production alert and pager rules>
  roles:
    facilitator: <name>        # injects failures, describes what responders would see
    responders: [<on-call-engineer>, <customer-side-owner>]
    scribe: <name>             # timestamps every event
    observer: <project-manager>
  scenarios:
    - name: dependency down
      inject: scale the dependency to zero replicas
    - name: credential expired
      inject: replace the secret with a bad value, restart the pods
    - name: failover
      inject: switch traffic to the secondary instance or region
  measure: [time_to_detect, time_to_recover]
  pass_if:
    detect_within: <minutes, from the launch plan>
    recover_within: <minutes, from the launch plan>
    runbook_followed_without_help: true

This is the whole plan. Write the pass criteria before the day so nobody grades their own homework. If your launch plan has no target for detection time, the rehearsal has already found its first gap. Invite the customer-side owner too, because they inherit this system.

Use failure injection testing to break one thing on purpose

The simplest injection is to kill a dependency. On Kubernetes that is one command.

# Facilitator only. Run against the staging cluster, never production for a first rehearsal.
date -u '+%H:%M:%S injected'
kubectl scale deployment/<dependency> -n <namespace> --replicas=0

# Safety net: restore only if the team has not recovered it themselves.
kubectl scale deployment/<dependency> -n <namespace> --current-replicas=0 --replicas=<original-count>

The first command records the injection time and scales the dependency to zero. The second restores it, and --current-replicas=0 makes it run only if the deployment is still at zero (see the kubectl scale reference). Check the original replica count before you start.

The scribe then records three times: injected, detected and recovered. Don't coach. The facilitator describes what responders would see on a dashboard, not what to do about it.

Answer the "we have no time" objection

The strongest objection is fair: launch is close, the plan is full, and a rehearsal costs half a day of several people. It may also find something that moves the date.

But the failure it finds will happen anyway. The only choice is whether it happens in a room with a facilitator or in front of customers. If half a day is too much, run the first scenario alone. For a small launch, that is a full rehearsal.

A rehearsal that finds nothing still pays off. You get measured detect and recover times to put in the launch sign-off, in place of "it should be fine".

Schedule your first rehearsal this week

Book 90 minutes with three to five people in the week before go-live. Run the dependency-down scenario first. Afterward, fix the gaps, tick the gate on your go-live checklist, and pick a different failure for the next launch.

For anyone putting AI into production, the model API, the vector store and the keys that guard them are all dependencies that will fail, and it is far cheaper to learn who notices on a Tuesday afternoon in staging than on launch night.