Fail CI on Flaky Playwright Tests Without Hiding Evidence

Here is the run that starts this problem. A pull request is green. The check is green, the merge button is green, and in the terminal output, forty lines above the summary, sits this:

  1 flaky
    example.spec.ts:5:2 › second flaky
  2 passed (4s)

Nobody read it, because nothing asked them to. The exit code was zero, GitHub showed a tick, and the reviewer looked at the diff. Six weeks later the same test fails on the first attempt and all three retries, someone opens the trace, and the failure turns out to be a race the team has been paying for since a config change in March. The signal was never missing. It was emitted, recorded, and discarded on every single run.

What Playwright actually means by "flaky"

The retries documentation defines three categories precisely, and the precision matters:

• passed — tests that passed on the first run.
• flaky — tests that failed on the first run, but passed when retried.
• failed — tests that failed on the first run and failed all retries.

So "flaky" in Playwright is not a heuristic, a score, or a judgement about a test's character. It is a mechanical fact about one execution: at least one attempt was red, and a later attempt was green. A test can be labelled flaky today and passed tomorrow, and both labels are correct records of what happened.

Two different properties encode this, and conflating them is the root of most broken flake tooling.

testResult.status is the status of one attempt: 'passed' | 'failed' | 'timedOut' | 'skipped' | 'interrupted'. Each retry produces its own TestResult with its own status and its own retry index, and they all live in testCase.results.

testCase.outcome() is the verdict over all attempts: 'skipped' | 'expected' | 'unexpected' | 'flaky'. The documentation is explicit that outcome is not the same as testResult.status, and gives the two cases that prove it: a test expected to fail that does fail is 'expected', and a test that passes on a second retry is 'flaky'.

"Expected" is doing real work there. testCase.expectedStatus is 'passed' for ordinary tests, 'skipped' for anything marked test.skip() or test.fixme(), and 'failed' for anything marked test.fail(). Outcome is the comparison of actual against expected, not the raw actual. This is why "did the test pass" is the wrong question to ask a reporter, and outcome() === 'flaky' is the right one.

Inside a test, the same fact is available as testInfo.retry: zero on the first run, one on the first retry, and so on. It is readable from any test, hook or fixture.

Where the signal surfaces, and exactly where it dies

Playwright surfaces flakiness in every built-in reporter. The list reporter prints a flaky block with the test titles. The dot reporter has a dedicated character for it: ± means "passed on retry (flaky)", distinct from ×, which means "failed or timed out — and will be retried". The HTML report keeps the failed attempt alongside the passing one.

The JSON reporter carries it as structured data, in two places. Per test, JSONReportTest.status is one of 'skipped' | 'expected' | 'unexpected' | 'flaky'. And at the top of the document, already aggregated for you:

{
  "stats": {
    "startTime": "2026-03-11T08:14:02.118Z",
    "duration": 412803,
    "expected": 611,
    "unexpected": 0,
    "flaky": 9,
    "skipped": 4
  }
}

Now the part that decides everything. In Playwright's own runner, the ok flag on a spec is computed as test.outcome() === 'expected' || test.outcome() === 'flaky'. Flaky counts as ok. And testCase.ok() is documented as "whether the test is considered running fine. Non-ok tests fail the test run with non-zero exit code."

That is the whole failure. Playwright classified the run correctly, wrote nine flaky tests into a machine-readable file, and then exited zero because flaky is ok. CI reads the exit code. Nothing in the default pipeline reads stats.flaky. The evidence exists at full fidelity and has no consumer.

Gate on flake rate, not on any single flaky test

Since v1.52 there is a blunt instrument for this: failOnFlakyTests in the config, or --fail-on-flaky-tests on the command line. It exits with an error if any test is marked flaky.

// playwright.config.ts — the blunt version
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  failOnFlakyTests: !!process.env.CI,
});

On a small, healthy suite this is the right answer and you should stop reading this section. On a suite of 600 browser tests hitting a real backend it is not, for a reason that is arithmetic rather than ideology. If a single test has a genuine 0.5% first-attempt failure rate from causes outside your test code — a slow CDN response, a container that took an extra second to become healthy — then across 600 tests you expect roughly three flaky labels per run. failOnFlakyTests turns that into a red build on most runs, and a check that is red most of the time gets ignored or bypassed within a fortnight. You have moved the blindness, not removed it.

The gate that survives contact with a large suite is a rate gate with a threshold you chose deliberately, plus a named-test gate for repeat offenders. Rate is a property of the suite; individual flaky labels are noise on top of it.

Start by keeping the raw evidence. Reporters compose, so add JSON without giving up readable terminal output:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Retries buy evidence: a second data point on whether the failure reproduces.
  // They do not buy a green build — the gate below decides that.
  retries: process.env.CI ? 2 : 0,

  // Deliberately NOT failOnFlakyTests. The rate gate owns this decision.
  failOnFlakyTests: false,

  use: {
    // Keeps the failed attempt's trace even though a later retry passed.
    trace: 'retain-on-failure',
  },

  reporter: process.env.CI
    ? [
        ['github'],
        ['html', { open: 'never' }],
        ['json', { outputFile: 'flake-evidence/results.json' }],
      ]
    : 'list',
});

Then post-process. This script reads nothing but the JSON report, so it works unchanged whether the report came from one machine or from merged shards:

// scripts/flake-gate.mjs
// Usage: node scripts/flake-gate.mjs flake-evidence/results.json
import { readFileSync } from 'node:fs';

const THRESHOLD_PCT = 1.5;                 // suite-wide budget
const REPEAT_OFFENDERS = new Set([         // known-bad, individually gated
  'checkout.spec.ts:88:5 › applies a promo code',
]);

const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const { expected, unexpected, flaky, skipped } = report.stats;

// Skipped tests never ran, so they do not belong in the denominator.
const executed = expected + unexpected + flaky;
const rate = executed ? (flaky / executed) * 100 : 0;

// Walk the suite tree to name the offenders — stats alone cannot.
const names = [];
const walk = (suite) => {
  for (const spec of suite.specs ?? [])
    for (const t of spec.tests)
      if (t.status === 'flaky')
        names.push(`${spec.file}:${spec.line}:${spec.column} › ${spec.title}`);
  for (const child of suite.suites ?? []) walk(child);
};
report.suites.forEach(walk);

console.log(`flake rate ${rate.toFixed(2)}% (${flaky}/${executed}), budget ${THRESHOLD_PCT}%`);
for (const n of names) console.log(`  flaky: ${n}`);

const overBudget = rate > THRESHOLD_PCT;
const offender = names.find((n) => REPEAT_OFFENDERS.has(n));

if (offender) {
  console.error(`gate failed: quarantined test flaked in the main lane — ${offender}`);
  process.exit(1);
}
if (overBudget) {
  console.error(`gate failed: flake rate ${rate.toFixed(2)}% exceeds ${THRESHOLD_PCT}%`);
  process.exit(1);
}
console.log(`gate passed; skipped: ${skipped}`);

Note the two exits are different failures with different owners. Over-budget is a suite health problem for the team. A quarantined test flaking in the main lane means quarantine leaked, which is a process problem.

If you would rather the decision live inside Playwright than in a separate step, a custom reporter can override the run's exit status directly. onEnd is documented as being allowed to override the status and hence affect the exit code, by returning an object with a status field:

// reporters/flake-budget.ts
import type { FullResult, Reporter, Suite } from '@playwright/test/reporter';

class FlakeBudget implements Reporter {
  private suite!: Suite;
  constructor(private options: { maxPercent: number } = { maxPercent: 1.5 }) {}

  // Returning false lets Playwright keep a normal terminal reporter alongside.
  printsToStdio() { return false; }

  onBegin(_config: unknown, suite: Suite) { this.suite = suite; }

  async onEnd(result: FullResult) {
    const tests = this.suite.allTests();
    const executed = tests.filter((t) => t.outcome() !== 'skipped');
    const flaky = executed.filter((t) => t.outcome() === 'flaky');
    const pct = executed.length ? (flaky.length / executed.length) * 100 : 0;

    for (const t of flaky) {
      const attempts = t.results.map((r) => r.status).join(' → ');
      console.log(`FLAKY ${t.titlePath().join(' › ')} [${attempts}]`);
    }

    if (result.status === 'passed' && pct > this.options.maxPercent) {
      console.error(`flake budget blown: ${pct.toFixed(2)}%`);
      return { status: 'failed' as const };
    }
  }
}
export default FlakeBudget;

The t.results.map(r => r.status) line is the useful part: it prints the attempt sequence, so timedOut → passed and failed → failed → passed stop looking like the same event. They are not. The first is usually an environment or waiting problem; the second is usually a genuine race.

Wire whichever version you chose into CI so that the report is produced even when tests fail:

# .github/workflows/e2e.yml
- name: Run end-to-end tests
  run: npx playwright test

- name: Flake gate
  if: always()
  run: node scripts/flake-gate.mjs flake-evidence/results.json

- name: Keep the evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: flake-evidence
    path: |
      flake-evidence/results.json
      playwright-report/
    retention-days: 30

if: always() on the upload step is not decoration. Without it, the artifact upload is skipped precisely on the runs where the evidence mattered.

Quarantine, done honestly

A test that flakes repeatedly should leave the blocking lane. The dishonest version of that is test.skip() with a Jira ticket in a comment, which is indistinguishable from deletion after the second sprint: it no longer runs, no longer reports, and no longer appears in any count except stats.skipped.

Honest quarantine keeps the test running, keeps it visible, and keeps it attached to a person. Tags are the mechanism, because a tag is queryable from the CLI and shows up on testCase.tags.

// checkout.spec.ts
import { test, expect } from '@playwright/test';

// Wrong: silent, unowned, permanent.
// test.skip('applies a promo code', async ({ page }) => { /* ... */ });

// Right: still runs, in its own lane, with an owner and an expiry.
test('applies a promo code', {
  tag: '@quarantine',
  annotation: [
    { type: 'quarantine-owner', description: 'checkout-team' },
    { type: 'quarantine-until', description: '2026-04-15' },
    { type: 'issue', description: 'https://example.com/issues/4471' },
  ],
}, async ({ page }) => {
  // ...
});

Tags must start with @, and they can also be written as @-tokens in the title, which testCase.tags extracts. The details object is clearer for anything you intend to read programmatically.

Then split the lanes. The blocking job excludes the tag; a second, non-blocking job runs only the tag:

# Blocking lane: quarantined tests cannot break the merge.
npx playwright test --grep-invert @quarantine

# Observation lane: runs the quarantined tests, allowed to fail the job's
# status without failing the merge. Extra retries here are pure data.
PLAYWRIGHT_JSON_OUTPUT_NAME=quarantine.json \
  npx playwright test --grep @quarantine --retries=4 --repeat-each=5 \
  --reporter=json

The observation lane is where quarantine earns its keep. Five repetitions with four retries each gives a real failure rate for that test, rather than the single bit of information a normal run produces. Feed its stats into the same script and you have a per-test number to argue from.

The expiry matters more than the tag. Add a check that reads the annotations and fails if a quarantine-until date has passed. Without it, quarantine is test.skip() with extra steps.

Evidence worth keeping

A quarantined test can only be fixed from the failing attempt, and the failing attempt is the one your defaults are most likely to discard.

The trace modes differ in exactly this respect, and the distinction is documented:

• 'on-first-retry' — records a trace only for the first retry. The original failure is not captured. If the retry passes, you have a trace of a successful run, which is the least useful artefact available.
• 'retain-on-failure' — records a trace for every run but keeps it only for runs that failed, and a failed run's trace is kept even when a later retry passes. This is the mode that preserves flake evidence.
• 'retain-on-failure-and-retries' — keeps a trace for any run that failed or that is a retry, so you get the failure and the passing retry side by side. Larger artefacts, best diffing.

Video accepts the same modes. Screenshots use a different set: 'off' | 'on' | 'only-on-failure' | 'on-first-failure'.

Beyond artefacts, attach the environment facts a trace cannot contain. testInfo.retry tells you when you are on a retry, so you can label the attempt and capture backend state that will otherwise be gone:

// fixtures/evidence.ts — import this file from your spec files, or add it
// to the project's setup so the hook applies suite-wide.
import { test } from '@playwright/test';

test.afterEach(async ({ page }, testInfo) => {
  if (testInfo.status === testInfo.expectedStatus && testInfo.retry === 0) return;

  await testInfo.attach(`attempt-${testInfo.retry}-context.json`, {
    contentType: 'application/json',
    body: Buffer.from(JSON.stringify({
      attempt: testInfo.retry,
      status: testInfo.status,
      expectedStatus: testInfo.expectedStatus,
      durationMs: testInfo.duration,
      url: page.url(),
      buildSha: process.env.GITHUB_SHA,
      workerIndex: testInfo.workerIndex,
    }, null, 2)),
  });
});

Comparing testInfo.status against testInfo.expectedStatus rather than checking for 'failed' is what makes this correct for tests marked test.fail(), whose expected status is 'failed'. Attachments land in testResult.attachments and appear in the HTML report against the specific attempt that produced them.

If you shard, use the blob reporter and merge. Blob reports contain all the details about the test run and exist to make sharded reports mergeable; running your gate against per-shard JSON gives you per-shard rates that are individually too small to be meaningful.

Three ways a flake gate goes wrong

The denominator drifts. A rate expressed against a test count changes when someone adds 200 tests, and the same absolute number of flaky runs suddenly looks healthier. If your suite size moves a lot, gate on both: a percentage and an absolute ceiling on flaky count.

Serial mode inflates the count. In test.describe.serial(), all tests in the group are retried together. A single unstable test at position two causes the whole group to re-run, and every test in it that failed then passed is legitimately labelled flaky. One root cause, five flaky labels. Look at whether your flaky list clusters inside serial blocks before concluding the suite is degrading.

The gate is advisory. A job that prints a warning and exits zero is read for two weeks and ignored thereafter. The gate has to be able to turn the build red, on a rate the team agreed to, or you have rebuilt the original problem with more YAML.

Apply this now

Add ['json', { outputFile: 'flake-evidence/results.json' }] to your CI reporter array and upload it as an artifact with if: always(). Do not add a gate yet. Run for a week and read stats.flaky against stats.expected + stats.unexpected + stats.flaky on each run.

That gives you your suite's actual flake rate, which is the number you need before choosing a threshold. Set the budget slightly below the observed rate so the gate has something to do on day one, tag the two or three tests responsible for most of the count into @quarantine with an owner and an expiry date, and switch trace to 'retain-on-failure' so the next flake arrives with the failing attempt attached.

Frequently asked questions

Does a flaky test make Playwright exit non-zero?

No, not by default. Flaky counts as ok, and only non-ok tests fail the run with a non-zero exit code. Set failOnFlakyTests (or pass --fail-on-flaky-tests) if you want any flaky result to fail the run, or compute a rate yourself from the JSON report.

Should we just set retries: 0 instead?

That trades one blind spot for another. With no retries you cannot tell an intermittent failure from a deterministic one without re-running by hand, and the first data point about reproducibility is exactly what a retry gives you. Keep retries, and stop treating a passing retry as the end of the matter.

Is flake rate the same as first-attempt failure rate?

No. Flake rate counts only tests that failed and then passed. A test that fails all its retries is unexpected, not flaky, and never enters the flaky count no matter how unstable it is. If you want first-attempt failure rate, count results with retry === 0 and a status other than 'passed' across testCase.results.

Can I give one unstable file more retries without changing the global config?

Yes. test.describe.configure({ retries: 2 }) sets retries for a describe group or a single file. Treat it as a declaration that the file is unstable, not as a fix, and pair it with the same visibility you would apply anywhere else.

Why does a test flake on CI but never locally?

Usually because CI runs more tests concurrently against shared external state. Playwright discards the entire worker process and its browser after a failure and starts a new one, so browser state is not the cause. Look at what your tests share outside the browser: seeded rows, fixed ports, accounts, files.

Primary references

• Playwright — Retries: the passed/flaky/failed categories, testInfo.retry, per-describe retries, and serial-mode group retry behaviour
• Playwright — Reporters: composing multiple reporters, the dot reporter's ± character, JSON and blob reporter options, and the custom reporter skeleton
• Playwright — TestCase: outcome() versus ok(), expectedStatus, results, and tags
• Playwright — TestResult: the per-attempt status values, retry index, and attachments
• Playwright — Reporter: onEnd overriding the run status and exit code, and printsToStdio
• Playwright — TestConfig: failOnFlakyTests, added in v1.52
• Playwright — Test use options: trace, video and screenshot modes, and which of them retain a failed attempt's artefacts after a passing retry
• Playwright — Annotations: tag syntax and filtering with --grep and --grep-invert