Flaky tests and CI test failures

Find the flake, then classify it

Both tools run in your browser and upload nothing.

Failure guides by cause

Each guide starts from the exact error string, then ranks the causes and gives the check that tells them apart. The full set is in the failure pattern library.

Playwright Reliability Lab, in order

The series this hub draws on, part by part.

1. Part 1Playwright Test Architecture: Start with Risk, Not Pages

   Page objects make every screen equally easy to test, and equal ease produces equal coverage of things with wildly unequal consequences. This shows how to score capabilities by impact multiplied by likelihood, then encode that ranking in Playwright projects, tags, boxed steps and per-tier retries so the risk model actually changes what runs.

2. Part 2Playwright Auto-Waiting vs Business-State Waiting

   Playwright's six actionability checks tell you an element is ready to click. They say nothing about whether the order was placed. Most flaky waits are that second question answered with the first question's tool.

3. Part 3Debug Playwright Strict-Mode Locator Failures

   Strict mode is Playwright refusing to guess which element you meant. The error already lists the candidates and suggests a fix for each. Here is the order to work through, and why an index is almost never the right answer.

4. Part 4Design Playwright Fixtures for Parallel Test Isolation

   Going parallel does not break tests; it reveals the ones that were already sharing state. Here is how to choose fixture scope deliberately, and why parallelIndex and workerIndex are not interchangeable.

5. Part 5Playwright Trace Viewer as a Root-Cause Workflow, Not a Video Player

   A Playwright trace is a recorded crime scene, not a screencast. This walks through what each panel proves, a seven-step read order for a CI-only failure, and why the standard on-first-retry setting often hands you a recording of the run that passed.

6. Part 6Mock or Integrate? Playwright Network Testing Boundaries

   Mocking is a decision about which contract a test asserts on, not a shortcut for speed. This guide gives you the criterion for mocking versus integrating, the exact differences between fulfill, continue, fallback, abort and fetch, and a concrete way to catch mocks that have drifted away from the real API.

7. Part 7Fail CI on Flaky Playwright Tests Without Hiding Evidence

   Playwright labels a test that passes on retry as flaky, counts it as ok, and exits zero — so the build goes green and the signal is discarded. This shows you how to read stats.flaky out of the JSON report, gate CI on a flake rate rather than on any single flaky test, and quarantine with tags, owners and expiry dates instead of test.skip().

8. Part 8Reviewing AI-Generated Playwright Tests: Eight Tells in the Diff

   A model writing Playwright reaches for the same constructs every time, so reviewing generated specs is a short checklist rather than an audit — and every item is visible in the diff without running anything. The eight tells: waitForTimeout standing in for a wait condition, an await inside expect() that kills retrying, .first() silencing a strict-mode violation, assertions that cannot fail, an over-broad page.route(), module-level state that breaks under fullyParallel, hardcoded URLs that belong in baseURL and storageState, and toHaveScreenshot() baselines generated on someone's laptop. It ends with the greps, lint rules and repeat-each run that make most of the pass mechanical.

More on flaky tests and CI failures

• Audit Test Names Before They Reach CI

  A CI failure named 'checkout works' hides the failing behavior. Check a batch of names against one convention, then compare each rename with the assertion it describes.

• Create a Failure Triage Toolkit Before the Next Flake

  "Flaky" is the label teams reach for when the run produced no evidence to say anything more precise. This is the Playwright artifact configuration, diagnostic fixture, reporter and CI wiring that make the next failure classifiable into product bug, test bug, environment, data, or genuine non-determinism.

• Debug JUnit Parallel-Execution Flakes

  A test that passes alone and fails in the suite, with the failure moving between classes, is an ownership bug that parallel execution revealed rather than caused. This guide covers pinning the configuration so the flake reproduces, the documented reason your fork-join pool can run more tests at once than the parallelism you configured, a probe that shows which threads two failing tests were on, when the answer is @TempDir rather than a lock, what @ResourceLock's READ and READ_WRITE modes actually schedule, and the check that tells Jupiter's threads apart from Gradle's forked JVMs.

• Debug Vitest Module Mocking and Cache Leaks

  When a Vitest test passes with -t and fails in the full file, a module cached by an earlier test usually holds the state. This guide reproduces the failure with a fixed shuffle seed, shows what per-file isolation covers in each pool (including process.env surviving between files in the vm pools), fixes an import-time environment read with vi.stubEnv, a scoped vi.resetModules and onTestFinished, and separates late mocks, module singletons and leaked fake timers by their symptoms. It also shows why clearMocks, mockReset, restoreMocks and unstubEnvs leave the module cache alone.

• Design pytest Fixtures as Explicit Resource Contracts

  A test that passes alone and errors in the full suite is almost never a flaky test — it is a fixture whose lifetime is wider than the state it holds, requested by nothing the test names. This guide works from what pytest actually reads when it orders fixtures, why one autouse fixture pulls its whole dependency graph into every test in scope, and why teardown after yield covers only the fixtures that already succeeded.

• Find Flaky Tests by Comparing JUnit XML Runs

  A single JUnit report can record a failed-then-passed test as a plain pass. Three real Playwright builds show what each producer writes for a retry and how comparing runs exposes the flake.

• Make JUnit Failures Useful in CI

  A red CI job that prints AssertionFailedError and a line number forces the next engineer to rebuild the scenario by hand. Using a parameterized contract test for GET /orders/{id}, this guide covers why Gradle's LIFECYCLE console omits assertion messages, keeping timestamps and generated ids out of test names, path/expected/actual messages with Supplier and assertAll, TestReporter entries and sanitized file attachments that Gradle 9.4 now carries into HTML and XML, redacting before anything reaches stdout, and why mergeReruns can make a flaky failure read as an ordinary pass.

• Playwright MCP Is Not Your Regression Test Suite

  A Playwright MCP session confirmed a checkout coupon worked, and the assertion it recorded passed against the correct page and against three broken versions of it. This guide runs @playwright/mcp 0.0.83 and Playwright 1.64.1 on a loopback lab: what the accessibility snapshot shows, why the agent's default browser differs from your CI build, how a verified text becomes a bare visibility check, and why an agent working at its own pace misses a sub-second race. It then encodes the race with a held page.route, proves each test against three hand-written mutants, and sets limits for Playwright's planner, generator and healer agents.

• Run pytest in Parallel Without Shared-State Flakes

  Adding workers does not create shared state — it reveals that the suite only ever worked because it ran one test at a time. This guide starts from the fact that every xdist worker collects independently and runs its own copy of each session-scoped fixture, then covers deriving a reclaimable namespace from worker_id, one PostgreSQL schema per worker, why collection order has to agree across workers, and why -s cannot work under xdist at all.

• Triage a Playwright Strict Mode Error Before Editing the Locator

  A Playwright strict mode error means a locator matched multiple elements. Inspect the matches before changing the test.

• Use pytest Parametrization Without Hiding Test Intent

  The generated ID is the only part of a parametrized case that reaches the report, the selection flag and the failure history — so an auto-generated one costs you both diagnosis and reproduction. This guide covers the exact rule pytest uses to build IDs, the default that silently renumbers collisions into positions, the effects that fire at collection time because a parameter table is just an expression in a decorator, and how to choose equivalence classes instead of Cartesian products.