How to Measure Coding Agent Impact Without Vanity Metrics

Your dashboard says developers accepted thousands of agent suggestions last month. Did anything ship faster, or did reviewers just inherit more diffs? To measure coding agent impact, use numbers from the delivery side of the pull request, which gh and git can pull in an afternoon.

You will end with a frozen baseline, a script for PR cycle time and review time, a revert count, and a change failure rate, with agent-assisted PRs split out.

Why acceptance counts don't show AI coding productivity

Acceptance rate tells you someone pressed Tab. Lines generated rewards verbosity. Neither says how fast a change reaches production or whether it stays there.

Feeling fast and being fast differ. In METR's randomized trial, 16 experienced open-source developers took 19% longer with AI tools allowed, while believing they were about 20% faster (METR). The 2024 DORA report tied a 25% rise in AI adoption to an estimated 7.2% drop in delivery stability (InfoQ summary). Both used older tools, so treat them as a reason to measure your own, not as your answer.

Four numbers to measure coding agent impact

All four come from data GitHub already holds:

• PR cycle time: pull request opened to merged. Use the median.
• Review time: PR opened to the first review by someone other than the author. Bigger agent PRs show up here first.
• Rework: the share of merged changes reverted within 14 days. Fix the window now. This is a revert rate, narrower than DORA's deployment rework rate.
• Change failure rate: the ratio of deployments that require immediate intervention, per the DORA metrics guide.

Track lines changed per PR beside them, so you can tell faster from smaller.

Step 1: Freeze definitions and a baseline

Write down the four definitions and the 14-day window before anyone sees agent numbers, so nobody picks definitions that flatter the result. Your baseline is the quarter before rollout. GitHub keeps PR history, so you can pull it late, but do it before results arrive.

Label agent-assisted PRs agent-assisted. A CI step that checks for the agent's Co-Authored-By trailer or a branch prefix beats relying on memory.

Step 2: Pull PR cycle time and review time with gh

This writes one CSV row per merged PR. Swap in your repo and dates.

echo 'number,cohort,lines_changed,hours_to_first_review,hours_to_merge' > baseline.csv
gh pr list --repo <org>/<repo> --state merged --limit 1000 \
  --search 'merged:2026-04-01..2026-06-30 -author:app/dependabot' \
  --json number,author,labels,createdAt,mergedAt,additions,deletions,reviews \
  --jq '.[] | . as $pr
    | ([.reviews[] | select(.author.login != $pr.author.login) | .submittedAt] | sort | first) as $first_review
    | [ .number,
        (if any(.labels[]; .name == "agent-assisted") then "agent" else "other" end),
        (.additions + .deletions),
        (if $first_review then (($first_review | fromdateiso8601) - (.createdAt | fromdateiso8601)) / 3600 else null end),
        ((.mergedAt | fromdateiso8601) - (.createdAt | fromdateiso8601)) / 3600
      ] | @csv' >> baseline.csv

It skips Dependabot, ignores the author's own review comments when finding the first review, and sets --limit because gh pr list returns 30 PRs by default. Run it again for the post-rollout quarter into after.csv, then summarize both with this summarize.py:

import csv
import statistics
import sys
from collections import defaultdict

cohorts = defaultdict(list)
with open(sys.argv[1], newline='') as f:
    for row in csv.DictReader(f):
        cohorts[row['cohort']].append(row)


def median(rows, column):
    values = [float(r[column]) for r in rows if r[column]]
    return round(statistics.median(values), 1) if values else None


for name, rows in sorted(cohorts.items()):
    print(
        f"{name}: {len(rows)} PRs, "
        f"median hours to first review {median(rows, 'hours_to_first_review')}, "
        f"median hours to merge {median(rows, 'hours_to_merge')}, "
        f"median lines changed {median(rows, 'lines_changed')}"
    )

Run python summarize.py baseline.csv, then the same with after.csv.

Step 3: Count reverts within 14 days

git fetch origin main
WINDOW_DAYS=14
git log origin/main --since='2026-07-01' --until='2026-10-01' \
  --grep='This reverts commit' --format='%H %ct' |
while read -r revert_sha revert_ts; do
  orig=$(git show -s --format=%B "$revert_sha" \
    | grep -oE 'This reverts commit [0-9a-f]{40}' | head -1 | awk '{print $4}')
  [ -z "$orig" ] && continue
  orig_ts=$(git show -s --format=%ct "$orig" 2>/dev/null) || continue
  age_days=$(( (revert_ts - orig_ts) / 86400 ))
  [ "$age_days" -le "$WINDOW_DAYS" ] && echo "$revert_sha reverted $orig after ${age_days}d"
done

Each default git revert message names the commit it undoes, so the loop compares the two commit dates. Run it per window, count the output lines, and divide by merged PRs in the same period for your rework rate. If your team reverts another way, adjust the grep or label revert PRs and count by label.

Step 4: Track change failure rate

If your CD tool records deployments and rollbacks, use those: that is DORA's definition. Without that data, label PRs that fix production problems hotfix and use them as a proxy.

merged=$(gh pr list --repo <org>/<repo> --state merged --limit 1000 \
  --search 'merged:2026-07-01..2026-09-30 -author:app/dependabot' --json number --jq 'length')
hotfix=$(gh pr list --repo <org>/<repo> --state merged --limit 1000 --label hotfix \
  --search 'merged:2026-07-01..2026-09-30 -author:app/dependabot' --json number --jq 'length')
awk -v h="$hotfix" -v m="$merged" 'BEGIN { print 100 * h / m "%" }'

The proxy holds when roughly one merge equals one deploy. If you batch deploys, count deployments instead.

Step 5: Read the four numbers together

• Cycle time down, review time flat, rework and failure rate flat or down: a real gain.
• Cycle time down, but review time, rework or failure rate up: you moved the cost onto reviewers or on-call.
• Nothing moved: check how many merges are agent PRs and what tasks they get.

Rerun monthly, comparing the same team before and after rollout.

Gotchas

• Task selection. People give agents easy tasks, so agent vs other PRs flatters the agent. Treat team before/after as the main comparison.
• Draft PRs. createdAt includes draft time, so long-lived drafts inflate cycle time.
• Bot reviewers. Automated reviews count as reviews. Filter their logins in the jq, or time to first review will look instant.
• Small samples. A handful of PRs a week gives noisy medians. Widen the window rather than slicing finer.
• Individual targets. Report per team, never per developer, or people will game the numbers.

Freeze the definitions, pull the baseline, rerun monthly. For anyone running AI in production, these four numbers show whether an agent speeds delivery or only moves the work onto reviewers and on-call.