Why Mutation Testing Catches Weak AI-Written Tests

Your coding agent just opened a pull request full of new tests, and the coverage report says 100%. That number only proves the tests ran your code, not that they would notice if the code were wrong. Mutation testing for AI-written tests closes the gap: it breaks your code on purpose and checks whether any test complains.

Why coverage lies about AI-written tests

Agents are good at making a test go green and weaker at deciding what the right answer is. With only the implementation to read, the cheapest assertion is one the code already satisfies: is not None, isinstance(...), > 0. Those lines execute everything and pin down nothing.

Mutation testing makes a small edit to your source, like >= to > or + to -, then runs your tests. If one fails, the mutant is killed. If everything stays green, the mutant survived: a bug just walked past your suite.

A passing suite that catches nothing

A discount function and the tests an agent might write for it:

# src/billing/discounts.py
def apply_discount(total: float, loyalty_years: int) -> float:
    if total <= 0:
        return 0.0
    rate = 0.0
    if loyalty_years >= 5:
        rate = 0.10
    elif loyalty_years >= 2:
        rate = 0.05
    return round(total * (1 - rate), 2)

# tests/test_discounts.py
from billing.discounts import apply_discount

def test_returns_positive_float():
    result = apply_discount(100.0, 5)
    assert isinstance(result, float)
    assert result > 0

def test_no_loyalty_pays_full_price():
    assert apply_discount(100.0, 0) == 100.0

def test_mid_tier_discount_applies():
    assert apply_discount(100.0, 3) > 0

def test_zero_total():
    assert apply_discount(0, 3) == 0.0

Every line and branch runs, so coverage reads 100%. Four edits a mutation tool could make all survive (mutmut picks its own set, but the failure is the same kind):

| Edit | Bug it models | | --- | --- | | loyalty_years >= 5 to > 5 | Five-year customers lose the top tier | | rate = 0.05 to rate = 0.5 | Mid tier gets half off | | 1 - rate to 1 + rate | Discount becomes a surcharge | | round(..., 2) to round(..., 3) | Prices with fractions of a cent |

The tests only ask "is it a positive float?". The fix is exact expected values on both sides of every boundary:

import pytest
from billing.discounts import apply_discount

@pytest.mark.parametrize(
    ("total", "years", "expected"),
    [
        (100.0, 1, 100.0),
        (100.0, 2, 95.0),
        (100.0, 4, 95.0),
        (100.0, 5, 90.0),
        (19.99, 5, 17.99),
        (0, 5, 0.0),
    ],
)
def test_discount_tiers(total, years, expected):
    assert apply_discount(total, years) == expected

The year-1/2 and year-4/5 rows pin the tier boundaries, 95.0 and 90.0 pin the rates, and 19.99 pins the rounding. Each mutant above now fails a row.

Run mutation testing on AI code with mutmut

Install mutmut and point it at your source and tests:

pip install mutmut

# pyproject.toml
[tool.mutmut]
source_paths = [ "src/" ]
pytest_add_cli_args_test_selection = [ "tests/" ]

Then aim it at one module, not the whole repo:

mutmut run "billing.discounts*"
mutmut results
mutmut show <mutant-name>
mutmut browse

run takes wildcard patterns for a module or function, results lists mutants by status, show prints one mutant's diff, and browse opens a terminal UI. mutmut needs fork, so on Windows use WSL.

Paste a survivor's diff back to your agent with "write a test that fails on this change." That prompt is precise and checkable, which beats "improve the tests."

Keep it fast: mutate only what the PR changed

The fair objection is speed: every mutant needs a test run, and a full-repo pass on every pull request gets the check disabled. Three things keep it sane. mutmut maps tests to the functions they call, so each mutant runs only relevant tests. You scope the run to files the PR touched. The full-repo pass goes in a nightly job.

This GitHub Actions job mutates only changed modules and fails below a score floor. Adjust the src/ prefix and patterns to your layout:

name: mutation-check
on:
  pull_request:
    paths: ["src/**", "tests/**"]
jobs:
  mutmut:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -e . pytest mutmut
      - name: Mutate changed modules and enforce a score floor
        run: |
          set -f
          patterns=$(git diff --name-only --diff-filter=AM "origin/${{ github.base_ref }}...HEAD" -- 'src/*.py' \
            | sed -e 's#^src/##' -e 's#\.py$##' -e 's#/#.#g' -e 's#$#*#')
          if [ -z "$patterns" ]; then echo "No Python source changes"; exit 0; fi
          mutmut run $patterns
          mutmut export-cicd-stats
          python - <<'EOF'
          import json, sys
          s = json.load(open("mutants/mutmut-cicd-stats.json"))
          tested = s["killed"] + s["survived"]
          score = s["killed"] / tested if tested else 1.0
          print(f"mutation score: {score:.0%} ({s['killed']} killed, {s['survived']} survived)")
          sys.exit(0 if score >= 0.80 else 1)
          EOF

mutmut run exits non-zero on setup failures, not on survivors, so the job reads the stats file and applies its own floor. Start the floor where your first run lands, then raise it.

For JS and TS, Stryker has an incremental mode that reruns only mutants affected by changes, and thresholds.break sets the failing score. Install @stryker-mutator/core and @stryker-mutator/jest-runner, then:

{
  "testRunner": "jest",
  "mutate": ["src/**/*.ts", "!src/**/*.spec.ts"],
  "coverageAnalysis": "perTest",
  "incremental": true,
  "thresholds": { "high": 80, "low": 60, "break": 60 },
  "reporters": ["clear-text", "progress", "html"]
}

changed=$(git diff --name-only --diff-filter=AM origin/<base-branch>...HEAD -- 'src/*.ts' ':!src/*.spec.ts' | paste -sd, -)
[ -n "$changed" ] && npx stryker run --incremental --mutate "$changed"

Cache reports/stryker-incremental.json between CI runs, or incremental mode has nothing to reuse.

Where mutation testing is just noise

Some survivors are harmless. Changing total <= 0 to total < 0 behaves identically for a zero total, so no test can kill it. These equivalent mutants are why you shouldn't chase 100%: set a floor, skip noisy lines such as logging with # pragma: no mutate, and read survivors instead of counting them.

Try it on one module today

Point mutmut run "<module>*" at the module your agent changed most recently. Read the first three survivors with mutmut show, then get one test that kills each. If those tests find real gaps, add the CI step.

When an agent writes both the code and the tests, a mutation score is the cheapest check that your safety net would catch a bug before production.