Why You Should Quarantine Flaky Tests Instead of Retrying

The most dangerous button in your CI is the one labelled re-run. Press it on a red build and the build goes green, nobody learns why it was red, and whatever caused it moves one step closer to production.

Auto-retry is that button pressed by a script. Quarantine flaky tests instead: move them to a lane that never blocks a merge, give each one an owner and a deadline, and fail the build when the list grows.

Why retrying flaky tests backfires

It hides the signal. Test retries hide failures by design: a test that passes on its second attempt looks identical to one that passed on its first, so nobody sees the flake rate creeping up. Intermittent failures are often real races or timeouts in the code under test, and a retry buries the evidence.

It taxes everyone. Every retry reruns the test and its setup on a PR that did nothing wrong. The team pays in minutes while the test's owner pays nothing.

It teaches people that red is noise. Google's testing team reported that about 1.5% of its test runs were flaky and that about 84% of pass-to-fail transitions involved a flaky test. When most red results are noise, engineers learn to re-run first and read later. Then a real regression shows up, gets the same treatment, and merges.

Retries aren't evil: the same Google post lists automatic re-runs among its mitigations. The mistake is making retry the policy instead of a stopgap.

Build a lane to quarantine flaky tests in CI

Four rules: the test is tagged, it keeps running outside the merge gate, it has an owner and a deadline, and the list can only shrink. The examples use pytest and GitHub Actions, but any tag-aware runner works the same way.

Keep one list with an owner and an expiry date

# quarantine.yaml
max_entries: 10   # lower this as the list shrinks; raising it needs a reviewer
tests:
  - id: tests/integration/test_billing.py::test_invoice_sync
    owner: payments-team
    issue: PLAT-1423
    expires: 2026-11-19
    reason: shared database fixture race

Every entry needs a named owner, a tracking issue and an expiry date a sprint or two away.

Tag quarantined tests from that list

# conftest.py
import pathlib

import pytest
import yaml

QUARANTINED = {
    t["id"]
    for t in yaml.safe_load(pathlib.Path("quarantine.yaml").read_text())["tests"] or []
}


def pytest_configure(config):
    config.addinivalue_line(
        "markers", "quarantine: known-flaky test, runs in the non-blocking lane"
    )


def pytest_collection_modifyitems(config, items):
    for item in items:
        if item.nodeid in QUARANTINED:
            item.add_marker(pytest.mark.quarantine)

This registers a quarantine marker and applies it to every test on the list, so nobody edits test files. Add pyyaml to your test requirements.

Split CI into a blocking lane and a flaky test quarantine lane

# .github/workflows/ci.yml
name: ci
on: pull_request

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -m "not quarantine"

  quarantine-lane:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -m quarantine --junitxml=quarantine.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: quarantine-results
          path: quarantine.xml

The tests job gates merges and skips anything quarantined. The lane runs only quarantined tests, and continue-on-error: true lets that job fail without failing the workflow run. The JUnit file keeps every run's result, so you can tell whether a fix worked.

Fail the build when the list grows or expires

  # add under jobs: in the same workflow
  quarantine-guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python scripts/check_quarantine.py

# scripts/check_quarantine.py
import datetime as dt
import pathlib
import sys

import yaml

data = yaml.safe_load(pathlib.Path("quarantine.yaml").read_text())
tests = data["tests"] or []
today = dt.date.today()
errors = []

if len(tests) > data["max_entries"]:
    errors.append(f"{len(tests)} tests quarantined, cap is {data['max_entries']}")

for t in tests:
    if not (t.get("owner") and t.get("issue")):
        errors.append(f"{t['id']}: needs an owner and an issue")
    if dt.date.fromisoformat(str(t["expires"])) < today:
        errors.append(f"{t['id']}: expired on {t['expires']}, owner {t.get('owner')}")

for e in errors:
    print(e)
sys.exit(1 if errors else 0)

This job is blocking, and it turns quarantine from a graveyard into a queue. Once a date passes, every PR goes red until someone fixes the test, deletes it, or pushes the date in a reviewed change with a name on it.

Isn't quarantine just ignoring tests?

It's the strongest objection, and it is half right. While a test is quarantined, the code it covers isn't guarded at merge time. That is a real cost, which is why the cap is low and the deadline short.

A skipped or deleted test produces no data and has no owner. A retried test hides its failures from everyone. A quarantined test still runs on every PR, still records its result, and has a name and a date attached. It is the only option where the failure stays visible and someone is accountable, which is most of what CI pipeline reliability means in practice.

Gotchas

• Keep quarantine-lane out of your required status checks, or you have rebuilt the blocking gate.
• With an empty list, pytest -m quarantine exits with code 5 (no tests collected), so skip the job or treat 5 as success.
• Parametrized tests need the full node id, brackets included, in quarantine.yaml.

Take the first step this week

Pull last month's CI history, list the tests that failed and then passed on the same commit, and move the worst five into quarantine.yaml with real owners and dates. Then turn off auto-retry on the main lane.

If your CI is where evals, agent tests and deploy checks run, a red build nobody trusts is a release gate nobody is actually enforcing.