🧭 How toCloudIntermediate✨ AI-assisted

How to Build a Handover Checklist for an AI Service

WittyTech Daily··6 min read
#handover#checklist#github-actions#yaml

The handover call ends with everyone nodding. Three weeks later the customer's finance team asks who pays for the GPU nodes, and the answer lives in a chat thread nobody can find.

An AI service handover checklist fixes that by turning "we're basically done" into a file that CI can reject. By the end you'll have a handover.yaml with an owner and an evidence link for every item, plus a short Python script that stays red until each item is closed or deliberately waived.

1. Write the AI service handover checklist as YAML

Keep the list in the repo so it gets reviewed like code. One entry per thing the customer must handle without you. Every entry needs an owner (a named person at the customer, not you) and, once it's done, a pointer to proof.

# handover.yaml
service: <service-name>
customer: <customer-name>
items:
  - id: account-ownership
    title: Customer organization owns the cloud account; vendor has no admin role
    owner: <name>@<customer-domain>
    status: done
    evidence: https://<customer-wiki>/handover/account-ownership
  - id: budgets-alerts
    title: Budgets and alerts cover cloud and model API spend
    owner: <name>@<customer-domain>
    status: todo
    evidence: null
  - id: secret-rotation
    title: All keys and tokens rotated, stored only in the customer's secret store
    owner: <name>@<customer-domain>
    status: todo
    evidence: null
  - id: runbooks
    title: Runbooks for provider outage, quality drop and key rotation
    owner: <name>@<customer-domain>
    status: done
    evidence: docs/runbooks/
  - id: on-call
    title: Customer on-call rota covers the service
    owner: <name>@<customer-domain>
    status: waived
    waived_by: <name>@<customer-domain>
    reason: Business-hours support only until Q1; revisit at the 30-day review
  - id: eval-and-prompts
    title: Eval set and production prompt version live in the customer's repo
    owner: <name>@<customer-domain>
    status: todo
    evidence: null
  - id: vendor-access-removed
    title: Vendor accounts, roles and keys removed
    owner: <name>@<customer-domain>
    status: todo
    evidence: null

status is todo, done or waived. A waived item needs a name and a reason, so skipping on-call is a decision someone signed, not something that quietly happened.

2. Decide what counts as evidence for operational readiness

Evidence is something a stranger can open and check. Agree on it with each owner before you start, not on the last day. Good defaults:

  • Cloud account ownership transfer: an export of the admin principals showing only customer identities, and a note that root credentials and MFA devices now sit with the customer.
  • Budgets and alerts: the budget definition and a test alert that reached a customer mailbox. On AWS, AWS Budgets data refreshes up to three times a day, so set thresholds below the number that actually hurts.
  • Secret rotation: a ticket showing every key and token was rotated after your last copy was made, with the new values only in the customer's secret store.
  • Runbooks: the repo path, plus a note of who at the customer has run one.
  • On-call: a rota with customer names and a test page that one of them acknowledged.
  • Eval set and prompt versions: the eval set path, the git tag of the prompts in production, and the eval results at that tag.
  • Vendor access removed: an access review export dated after every other item closed.

If you can't point at it, it isn't done.

3. Add a script that fails when evidence is missing

The script prints one FAIL line per problem: missing fields, duplicate ids, done items without evidence, waivers without a reason. With --strict it also fails on any item still todo. It parses with yaml.safe_load from PyYAML, which doesn't build arbitrary Python objects.

#!/usr/bin/env python3
# scripts/check_handover.py: fail while handover.yaml has items without owners or evidence.
import sys
from pathlib import Path
from urllib.parse import urlparse

import yaml

REQUIRED = ('id', 'title', 'owner', 'status')
STATUSES = ('todo', 'done', 'waived')


def evidence_ok(value):
    if not isinstance(value, str) or not value.strip():
        return False
    return urlparse(value).scheme == 'https' or Path(value).exists()


def problems(item):
    name = item.get('id', '<no id>')
    for key in REQUIRED:
        if not item.get(key):
            yield f'{name}: missing {key}'
    status = item.get('status')
    if status and status not in STATUSES:
        yield f'{name}: status must be one of {STATUSES}'
    if status == 'done' and not evidence_ok(item.get('evidence')):
        yield f'{name}: done, but evidence is not an https link or an existing repo path'
    if status == 'waived' and not (item.get('waived_by') and item.get('reason')):
        yield f'{name}: waived items need waived_by and reason'


def main():
    strict = '--strict' in sys.argv
    args = [a for a in sys.argv[1:] if a != '--strict']
    path = args[0] if args else 'handover.yaml'

    data = yaml.safe_load(Path(path).read_text(encoding='utf-8')) or {}
    items = data.get('items') or []
    errors = [] if items else ['no items found']

    ids = [i.get('id') for i in items]
    errors += [f'duplicate id: {d}' for d in sorted({i for i in ids if i and ids.count(i) > 1})]

    for item in items:
        errors += list(problems(item))
        if strict and item.get('status') == 'todo':
            name = item.get('id')
            errors.append(f'{name}: still todo')

    for error in errors:
        print('FAIL', error)
    closed = sum(i.get('status') in ('done', 'waived') for i in items)
    print(f'{closed}/{len(items)} items closed')
    return 1 if errors else 0


if __name__ == '__main__':
    sys.exit(main())

Evidence passes if it is an https link or a path that exists in the checkout, which is why the runbook entry points at a repo folder. Run pip install pyyaml, then run it locally before you push.

4. Run the check in CI before every production handover

Pull requests get the normal check, so a bad edit goes red. Tags named handover-* get strict mode, so you can't cut the handover tag while anything is still open. Syntax is in the GitHub Actions workflow reference.

# .github/workflows/handover.yml
name: handover-check
on:
  pull_request:
    branches:
      - main
  push:
    tags:
      - 'handover-*'
jobs:
  checklist:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - run: pip install pyyaml
      - name: Check structure and evidence
        run: python scripts/check_handover.py handover.yaml
      - name: Require every item closed
        if: startsWith(github.ref, 'refs/tags/handover-')
        run: python scripts/check_handover.py handover.yaml --strict

Mark the job as a required status check on main in branch protection, and require a customer reviewer on changes to handover.yaml. Otherwise you are marking your own homework.

5. Walk the list with the customer and close it out

Flip items to done in pull requests as evidence appears, not in one heroic commit on the last day. In the final session, have each owner open their own evidence link, and read every waiver aloud.

Leave vendor-access-removed for last, and let the customer's owner complete it.

Gotchas

  • Evidence in your systems dies with your access. Store it in the customer's wiki or copy exports into the repo before you leave.
  • The script proves evidence is present, not true. The customer reviewer clicking through is the actual control.
  • Don't paste secrets into evidence. Link to the rotation ticket or redact the export.
  • Plan the final merge. Once your access is removed you can't push, so agree who merges the last pull request and creates the handover-* tag.

A checklist that CI can fail is the cheapest way to make sure the team that gets paged at 2 a.m. is the one that can actually fix the service.

← More in Cloud