Why Automated Change Approval Beats a Change Advisory Board

Picture a one-line prompt fix that stops a support bot from quoting a refund policy that doesn't exist. It's ready Tuesday. The Change Advisory Board (CAB) meets Thursday. This is the case for automated change approval: let the pipeline decide whether a change is safe, using checks that inspect the change, instead of a weekly meeting that can't.

Why a weekly board can't judge an AI release

A CAB reads a ticket that says "update system prompt to v14, swap embedding model." Nobody in the room can say whether answer quality dropped, because that lives in an eval suite, not on a slide. So the board approves on the requester's reputation, or on a rollback plan that is one sentence in the ticket.

Meanwhile the urgent fix waits, and teams learn to batch changes to dodge the queue. Bigger batches are harder to review and harder to roll back, which is the opposite of what the board was for.

What DORA found about change advisory boards

DORA's research on streamlining change approval links heavyweight, externally driven approval to worse delivery performance. It found no evidence that a more formal external review lowers change fail rates. Slow approval pushes teams toward larger, less frequent releases, which carry more risk. Adding process after an incident makes it worse by stretching lead times and batch sizes.

Its recommendation: peer review during development, backed by automation such as continuous testing, continuous integration and monitoring. The CAB survives as a place to coordinate between teams and make business trade-offs, not as a gate.

Build automated change approval into the pipeline

Every check a board member would do by squinting at a ticket becomes a job that fails loudly. This GitHub Actions workflow (any CI works) runs tests and evals, checks manifests against policy, then deploys to a protected environment:

name: release
on:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make test
      # your own eval runner: fail the build if answer quality drops
      - run: python evals/run_suite.py --fail-under 0.92

  policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check manifests against Rego policies
        run: |
          docker run --rm -v "$PWD":/project openpolicyagent/conftest \
            test k8s/ --policy policy/

  deploy:
    needs: [test, policy]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      # assumes cluster credentials and the Argo Rollouts kubectl plugin are set up
      - run: kubectl apply -f k8s/rollout.yaml
      - run: kubectl argo rollouts status <your-service> --timeout 20m

The test job fails the build if the eval pass rate drops; evals/run_suite.py stands in for whatever eval runner you use. The policy job runs Conftest over your manifests. The deploy job waits for both through needs, and environment: production lets you restrict which branches can deploy and add protection rules later.

A policy is a Rego rule. This one, saved as policy/rollout.rego, rejects a Rollout that pins an image to latest:

package main

deny contains msg if {
  input.kind == "Rollout"
  some container in input.spec.template.spec.containers
  endswith(container.image, ":latest")
  msg := sprintf("%s uses the latest tag", [container.name])
}

Each rule you add is one fewer question for a meeting.

Let a canary gate roll back automatically

The last gate is production itself. With Argo Rollouts, an analysis step queries your metrics during the canary and aborts when they go bad:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://<your-prometheus>:9090
          query: |
            sum(irate(http_requests_total{service="{{args.service-name}}",code!~"5.*"}[5m])) /
            sum(irate(http_requests_total{service="{{args.service-name}}"}[5m]))
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: <your-service>
spec:
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: http-success-rate
            args:
              - name: service-name
                value: <your-service>.<namespace>.svc.cluster.local

Per the Argo Rollouts analysis docs, a failed analysis aborts the rollout, returns traffic to the stable version and marks the rollout Degraded. An inconclusive result pauses it until someone resumes or aborts. kubectl argo rollouts status returns an error unless the rollout ends healthy, so the pipeline job goes red too. For an AI service, swap the HTTP query for a metric that tracks answer quality, such as a guardrail trigger rate your service exports.

The strongest objection: auditors want an approval

Fair, and the answer is that approval moves rather than vanishes. DORA says peer review meets segregation of duties, where someone other than the author approves the change, provided reviews and approvals are captured in your development platform. Require pull request reviews, turn on the environment's "Prevent self-review" setting, and each change leaves a record of who reviewed it, which checks passed and what the rollout did. Meeting minutes rarely match that.

The honest limit: some changes deserve a named human. A schema migration on regulated data is not a prompt tweak, and DORA warns against treating every change the same. Route those to an environment with required reviewers (up to six people or teams) and automate the rest.

Automate your first gate this week

Take the last ten changes your board approved and find the lowest-risk kind, probably prompt edits or config-only changes. Write the one gate that would have caught a bad one, usually an eval threshold in CI, and agree that changes passing it are pre-approved. ITIL 4 calls this change enablement, and its standard changes work the same way: low-risk, well understood, pre-authorized.

Then measure what DORA suggests: the share of changes reaching production without manual approval, and how long changes wait for approval. Let the board keep what it does well, which is coordinating across teams.

When prompts and models change daily, the only approval that keeps up is one that runs your evals and watches the rollout.