trace: 'retain-on-failure' rather than 'on-first-retry', attach console, network and retry metadata from an automatic fixture, and keep a per-run ledger of failures. Then the label becomes a finding instead of a shrug.The sequence is familiar. A pipeline goes red on a checkout test. Someone reruns it. It goes green. The thread ends with \"flaky, ignore\". Three weeks later the same test fails in the same place, the same rerun clears it, and nobody has learned anything in between.
\n\nThat loop persists for a structural reason, not a cultural one. By the time somebody looked, the only surviving evidence was a stack trace saying a locator timed out after 30 seconds. From that single artifact, \"flaky\" really is the most defensible claim available. The problem is upstream: the run did not produce enough evidence to support a more precise one.
\n\nWhat you observed is that a test failed once and passed on re-execution. That is compatible with at least five underlying causes, and four of them are bugs you can fix.
\n\nGenuine non-determinism — an uncontrollable race, an animation frame, a clock boundary — is real, but in most suites it is the smallest of the five categories. It only looks largest because it is the default label for everything undiagnosed. A suite where 12% of failures are labelled flaky and 0% anything else does not have a flakiness problem; it has an evidence problem. Closing that gap has to happen before the failure, because afterwards the worker is gone, the browser is gone, and the container has been reclaimed.
\n\nThis is the highest-leverage change, and the most commonly misconfigured setting in Playwright projects.
\n\nAlmost every published starter config uses trace: 'on-first-retry'. That mode records a trace on the first retry only — nothing during the original run. The artifact you end up with describes an execution that passed, or at best a second failure that may have failed differently. The failure you care about was never recorded.
Playwright documents seven trace modes, and the differences are not cosmetic:
\n\n| Mode | Records a trace on | Keeps the trace when |
|---|---|---|
'off' | never | — |
'on' | every run | always |
'retain-on-failure' | every run | that run failed |
'retain-on-first-failure' | first run only | the first run failed |
'retain-on-failure-and-retries' | every run | that run failed, or it is a retry |
'on-first-retry' | first retry only | always |
'on-all-retries' | every retry | always |
For triage you want a mode from the \"records on every run\" family. 'retain-on-failure' traces every run and discards traces for runs that passed: full evidence for failures, near-zero storage for green runs, at the cost of tracing overhead on passing tests.
Where that overhead is unacceptable, 'retain-on-failure-and-retries' beats 'on-first-retry', because it captures the original failure and keeps the retry. Diffing a failed run against its passing retry is the most direct way to separate a data problem from a timing problem, and it is impossible if you kept only one of them.
The trace is the centrepiece but not sufficient alone. Video captures what the browser painted between actions, where CSS transitions and third-party overlays live. Screenshots survive tooling that cannot open a trace. preserveOutput decides whether the raw output directory survives at all.
// playwright.config.ts\nimport { defineConfig } from '@playwright/test';\n\nexport default defineConfig({\n // Retries are for keeping the pipeline moving. They are not the diagnosis.\n retries: process.env.CI ? 2 : 0,\n\n outputDir: 'test-results',\n // 'always' (the default) keeps output for passing tests too and grows fast.\n // 'failures-only' keeps precisely the runs you will look at.\n preserveOutput: 'failures-only',\n\n use: {\n // Records every run, keeps the failures. Do not use 'on-first-retry' here.\n trace: {\n mode: 'retain-on-failure',\n snapshots: true, // DOM snapshot per action - the \"why did the locator\n // not match\" evidence. Defaults to true.\n screenshots: true, // builds the timeline filmstrip\n sources: false, // omit source files; they bloat the zip in CI where\n // the checkout is already available\n attachments: true,\n },\n video: 'retain-on-failure',\n screenshot: 'only-on-failure',\n },\n\n reporter: process.env.CI\n ? [['blob'], ['github'], ['./reporters/flake-ledger.ts']]\n : [['list'], ['html', { open: 'never' }]],\n});Two details there are load-bearing. retries: 0 locally combined with trace: 'on-first-retry' means a developer running the suite on their machine never produces a trace at all, because there is never a first retry; the mode above behaves identically in both environments. And snapshots is what makes the trace viewer's DOM inspector work — with it off you get an action log and screenshots but cannot query the DOM at the moment the locator failed, which is the fastest way to tell \"never rendered\" from \"rendered but covered by an overlay\". Both fail with the same timeout message.
Triage is a classification task, so it needs mutually exclusive categories and an evidence test for each. Without the evidence test, everyone classifies by intuition and the buckets drift.
\n\n| Bucket | Claim | Evidence that confirms it |
|---|---|---|
| Product bug | The application did the wrong thing | A non-2xx response, a console pageerror, or a DOM snapshot showing the app in a state its own spec forbids |
| Test bug | The application was fine; the test asserted wrongly | DOM snapshot shows the expected state present, but under a different accessible name, in a second matching node, or behind an overlay the test never dismissed |
| Environment | The infrastructure under the test failed | Failures cluster by workerIndex or by wall-clock window; a worker restart in the run log; DNS or connection-refused entries rather than HTTP statuses |
| Data | Two runs contended for the same record | Failure correlates with parallelIndex, or reproduces only when a specific other test runs concurrently, or the API returns a 409/422 on a fixture the test believes it owns |
| Non-determinism | A genuine, uncontrolled race | Everything above ruled out; failure point moves between runs; the failed and passing traces show identical inputs and different interleaving |
The last bucket is defined by exclusion on purpose. It is the residue, and if it is not small, the evidence is still incomplete.
\n\nThis trips up nearly every hand-written logging fixture. Playwright's page.on('requestfailed') fires only when the client could not get an HTTP response at all — DNS failure, connection refused, net::ERR_FAILED. A 404 or 503 is a successful exchange from the protocol's point of view, so it emits requestfinished instead.
A fixture listening only for requestfailed will therefore report a clean network on a run where the backend returned 500 to every call, and the test presents as \"timed out waiting for the results table\" — indistinguishable from flakiness, actually a product bug. Listen for responses and filter by status.
Action timeouts cover two very different situations with one message. Open the failing action in the trace viewer and check the DOM snapshot at that step:
\n\npointer-events: none, or detached and re-rendered mid-action. That is almost always a test bug or a product bug in the overlay, not non-determinism.Non-determinism, by definition, correlates with nothing. Everything else correlates with something: a worker, a shard, a time of day, a concurrently running test, a deploy. That is why the ledger below matters more than any single trace — correlation is a property of many runs and invisible in one.
\n\nTraces cover browser activity. They do not cover what your test knew and never recorded: which seeded account it used, which flags were on, which build of the API it talked to. Attach that yourself. testInfo.attach() takes either body or path, never both, and infers content type from the path when contentType is omitted (defaulting to text/plain for strings, application/octet-stream for buffers). Attachments appear in the HTML report and travel inside blob reports.
// fixtures/diagnostics.ts\nimport { test as base } from '@playwright/test';\n\ntype Diagnostics = { diagnostics: void };\n\nexport const test = base.extend<Diagnostics>({\n diagnostics: [async ({ page }, use, testInfo) => {\n const console_: string[] = [];\n const pageErrors: string[] = [];\n const httpErrors: string[] = [];\n const transportErrors: string[] = [];\n\n page.on('console', msg => {\n if (msg.type() === 'error' || msg.type() === 'warning')\n console_.push(`[${msg.type()}] ${msg.text()}`);\n });\n\n // Uncaught exceptions in page scripts. Not the same as console errors.\n page.on('pageerror', err => pageErrors.push(err.stack ?? err.message));\n\n // HTTP-level failures: these do NOT surface via 'requestfailed'.\n page.on('response', res => {\n if (res.status() >= 400)\n httpErrors.push(`${res.status()} ${res.request().method()} ${res.url()}`);\n });\n\n // Transport-level failures: DNS, refused connection, aborted request.\n page.on('requestfailed', req =>\n transportErrors.push(`${req.url()} ${req.failure()?.errorText}`));\n\n await use();\n\n // Only attach on an unexpected outcome. Attaching on green runs makes the\n // report unreadable and inflates blob artifacts.\n if (testInfo.status === testInfo.expectedStatus) return;\n\n await testInfo.attach('triage-context', {\n contentType: 'application/json',\n body: JSON.stringify({\n retry: testInfo.retry,\n status: testInfo.status,\n expectedStatus: testInfo.expectedStatus,\n workerIndex: testInfo.workerIndex,\n parallelIndex: testInfo.parallelIndex,\n durationMs: testInfo.duration,\n lastUrl: page.url(),\n buildSha: process.env.GIT_SHA ?? 'unknown',\n apiBase: process.env.API_BASE_URL ?? 'unknown',\n httpErrors,\n transportErrors,\n pageErrors,\n console: console_.slice(-50),\n }, null, 2),\n });\n\n // A DOM snapshot the trace cannot give you: the serialised markup at the\n // moment of failure, greppable without opening a viewer.\n await testInfo.attach('dom-at-failure.html', {\n contentType: 'text/html',\n body: await page.content().catch(() => '<!-- page already closed -->'),\n });\n }, { auto: true }],\n});\n\nexport { expect } from '@playwright/test';Three notes. { auto: true } is what makes it run for every test without anyone importing it — a diagnostic fixture you must remember to opt into will not be there on the run that matters. The page.content() call is guarded because a test that crashed the page throws on any further page call, and an exception in teardown replaces the real failure with a confusing one. The console buffer is truncated because a chatty SPA otherwise produces a multi-megabyte attachment per failure.
Playwright's report already distinguishes flaky (failed first, passed on retry) from failed (failed every attempt). That distinction is per-run. Triage needs it accumulated across hundreds of runs, keyed so you can ask what a failure correlates with. A reporter is the cheapest place to produce that: onTestEnd receives a TestResult carrying retry, status, duration, errors, attachments, workerIndex and parallelIndex.
// reporters/flake-ledger.ts\nimport { appendFileSync } from 'node:fs';\nimport type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';\n\nexport default class FlakeLedger implements Reporter {\n onTestEnd(test: TestCase, result: TestResult) {\n if (result.status === 'passed' && result.retry === 0) return;\n\n appendFileSync('test-results/ledger.ndjson', JSON.stringify({\n runId: process.env.GITHUB_RUN_ID ?? 'local',\n shard: process.env.SHARD_INDEX ?? '1',\n buildSha: process.env.GITHUB_SHA ?? 'local',\n titlePath: test.titlePath().join(' > '),\n location: `${test.location.file}:${test.location.line}`,\n project: test.parent.project()?.name,\n attempt: result.retry,\n status: result.status, // passed | failed | timedOut | skipped | interrupted\n durationMs: result.duration,\n startedAt: result.startTime.toISOString(),\n workerIndex: result.workerIndex,\n parallelIndex: result.parallelIndex,\n // First error message only, normalised so identical failures group.\n error: result.errors[0]?.message\n ?.split('\\n')[0]\n .replace(/\\d+/g, 'N')\n .slice(0, 300),\n artifacts: result.attachments.map(a => a.name),\n }) + '\\n');\n }\n}Note that 'timedOut' is a status distinct from 'failed'. Collapsing the two, which most homegrown dashboards do, destroys the ledger's most useful signal: a timeout was waiting for something that never happened, while an assertion failure got a wrong value promptly. Different investigations.
Normalising digits out of the error message is what makes grouping work. Without it, Timeout 30000ms exceeded and Timeout 15000ms exceeded are two distinct failures and no pattern emerges.
One caveat on parallelIndex: it runs from 0 to workers - 1 and is guaranteed unique only among workers running at the same time. It is a slot number, not a process identifier — use it to detect contention between concurrent tests, not to track a process across a run.
Artifacts sitting on a reclaimed machine are not evidence. Two things go wrong most often: the upload step is skipped because the test step failed, and sharded runs produce fragmentary reports nobody can merge.
\n\n# .github/workflows/e2e.yml\njobs:\n test:\n strategy:\n fail-fast: false # a red shard must not cancel the others,\n matrix: # or you lose their evidence too\n shard: [1, 2, 3, 4]\n steps:\n - uses: actions/checkout@v6\n - uses: actions/setup-node@v4\n with: { node-version: 20 }\n - run: npm ci\n - run: npx playwright install --with-deps\n - run: npx playwright test --shard=${{ matrix.shard }}/4\n env:\n GIT_SHA: ${{ github.sha }}\n SHARD_INDEX: ${{ matrix.shard }}\n\n # Runs even when the test step failed - which is the only time it matters.\n - uses: actions/upload-artifact@v4\n if: ${{ !cancelled() }}\n with:\n name: blob-report-${{ matrix.shard }}\n path: blob-report/\n retention-days: 30\n\n - uses: actions/upload-artifact@v4\n if: ${{ !cancelled() }}\n with:\n name: traces-${{ matrix.shard }}\n path: test-results/\n retention-days: 14\n\n report:\n needs: [test]\n if: ${{ !cancelled() }}\n steps:\n - uses: actions/checkout@v6\n - run: npm ci\n - uses: actions/download-artifact@v4\n with: { path: all-blobs, pattern: blob-report-*, merge-multiple: true }\n - run: npx playwright merge-reports --reporter=html ./all-blobsThe blob reporter exists for exactly this: it records full run detail and merge-reports reassembles the shards into one HTML report afterwards. Blob files are named report-<hash>-<shard_number>.zip, where the hash derives from the command-line filters, so two differently filtered runs cannot silently overwrite each other.
Once traces sit behind a URL you never need to download them. npx playwright show-trace https://.../trace.zip opens a remote trace directly, and the hosted viewer accepts the trace URL as a query parameter. Putting that link in the failure notification removes the largest single piece of triage friction: the download-and-unzip step that stops people from looking at all.
Each of these has a different cause and a different fix.
\n\nEverything is retained, so nothing is read. preserveOutput: 'always' plus trace: 'on' produces gigabytes per run and a report that takes a minute to load. People stop opening it. The fix is asymmetric retention: full fidelity for failures, nothing for passes.
Retries mask a real regression. Two retries turn a product bug that fires 40% of the time into a green pipeline roughly 78% of the time. The ledger is the defence: a test repeatedly marked flaky is a test repeatedly failing, and the ledger shows that even though every run was green.
\n\nThe diagnostic fixture becomes the flake. A fixture that awaits a network call, writes a large file, or attaches on every test adds its own timeouts and its own teardown failures. Cap the payload and swallow exceptions inside it.
\n\nClassification without a schema. If the bucket lives in a free-text ticket comment, nobody can count them. Put the five categories in a required field; the monthly distribution then answers \"where should we spend engineering time\" directly.
\n\nToday, in one commit: change trace from 'on-first-retry' to 'retain-on-failure', add video: 'retain-on-failure', set preserveOutput: 'failures-only', and confirm the artifact upload step carries if: ${{ !cancelled() }}. This week: add the automatic diagnostics fixture and the ledger reporter and let them run.
The evidence to capture per failure, before anyone is allowed to write \"flaky\": the trace zip from the failing run, the status of every non-2xx response during the test, the count of DOM nodes matching the failing locator at the moment it failed, and the ledger rows for that test over the last thirty days. If all four are unremarkable, the label has been earned.
\n\nretain-on-failure slow the suite down noticeably?It records on every run, so the overhead applies to passing tests too, dominated by DOM snapshotting per action. If you measure a real regression, keep the mode and disable sources first, then screenshots, before falling back to 'retain-on-failure-and-retries' — which, unlike 'on-first-retry', still captures a failing first run.
That surfaces every failure but stops the pipeline on each one, and it removes the passing-retry artifact that makes diffing possible. Keep retries for throughput and treat the flaky count as a first-class metric that must trend to zero. A retry is acceptable when it is recorded and reviewed, not as a substitute for review.
\n\ntestInfo.attach() rather than testInfo.outputPath()?Use attach() when a human should see the file in the report — it copies the file to a reporter-accessible location and registers it, and you may delete your copy once the promise resolves. Use outputPath() for scratch files the test itself consumes; those live in a per-test subdirectory of outputDir, which is what stops parallel tests colliding.
Environment. The signature is a requestfailed or a long-tail response on a domain you do not own, while your own API calls are clean. Route-block that domain in test runs once you have the evidence — but not pre-emptively, or you lose the ability to detect it.
trace with mode, snapshots, screenshots, sources and attachments.attach() semantics (body and path are mutually exclusive), retry, and the status versus expectedStatus comparison.onTestEnd, including parallelIndex, workerIndex and the distinct timedOut status.merge-reports for sharded runs.requestfailed excludes HTTP error responses such as 404 and 503.trace.playwright.dev query-parameter loading.outputDir per-test subdirectories and the three preserveOutput values.