A Playwright build finishes with one red test, rejects an expired card. The team fixes it and moves on. The same report also contains applies a coupon, which failed its first attempt, passed on retry, and appears in the XML as a passing <testcase> with no <failure> child. Nothing in that file says it was ever red. The only way to see it is to compare it with other builds, or to make the reporter keep the failed attempt.
There is no single JUnit XML standard. The Ant JUnit task made the format common; Maven Surefire publishes its own XSD; the rest follow the Ant shape loosely. What differs most, and matters most for flakes, is what happens to a failed attempt that is followed by a pass.
\n\n| Producer | Failed attempt, then a pass | Recorded time |
|---|---|---|
Maven Surefire and Failsafe with rerunFailingTestsCount | One <testcase> with a <flakyFailure> or <flakyError> per failed attempt; the XSD also defines a flakes count on the suite | The last successful run |
| Maven Surefire, every rerun fails | <failure> for the first run plus one <rerunFailure> per rerun | The first failing run |
| Gradle, default | Each execution is a separate <testcase> with the same name | Per execution |
Gradle with mergeReruns = true | One <testcase> with <flakyFailure>, the Surefire shape | Not stated in the manual |
Playwright junit reporter, default | A plain passing <testcase> | The sum of every attempt |
Two details in that table cause most misreadings. In Surefire's schema, a rerun's trace sits inside a <stackTrace> child element, while the first failure's trace is the <failure> element's own text, so a parser that only reads element text finds empty reruns. And Gradle warns that a CI server that does not understand <flakyFailure> \"will indicate that the test succeeded\". The manual names Jenkins' JUnit plugin, which leaves those failed executions out of its test result views.
Other producers differ in ways that affect matching rather than retries. pytest wraps a single <testsuite> in a <testsuites> root and, when a test fails in its call and then errors in teardown, writes two <testcase> elements for it. Its junit_duration_report setting defaults to total, so a test's time includes setup and teardown. Vitest names each case with its ancestor describe blocks joined by > ; jest-junit's default titleTemplate is {classname} {title}. Change either template between builds and the same test gets a new identity.
The reports below came from Playwright 1.62.1 with retries: 1. A BUILD variable stands in for a coupon service that is sometimes slow: build 1 fails the coupon test's first attempt only, build 2 fails both attempts, and build 3 never fails. The expired-card test is a real bug and fails every time.
// playwright.config.ts\nexport default defineConfig({\n retries: 1,\n reporter: [[\"junit\", { outputFile: `results/build-${process.env.BUILD}.xml` }]],\n})\n\n// tests/checkout.spec.ts\ntest(\"applies a coupon\", async ({}, testInfo) => {\n const fails = (build === 1 && testInfo.retry === 0) || build === 2\n await new Promise(resolve => setTimeout(resolve, fails ? 2000 : 300))\n expect(fails ? \"timeout\" : \"applied\", \"coupon status\").toBe(\"applied\")\n})for b in 1 2 3; do BUILD=$b npx playwright test; doneBuild 1's report records the flaky test like this:
\n\n<testcase name=\"checkout › applies a coupon\" classname=\"checkout.spec.ts\" time=\"2.306\">\n<system-out>\n<![CDATA[\n[[ATTACHMENT|../test-results/checkout-checkout-applies-a-coupon/error-context.md]]\n]]>\n</system-out>\n</testcase>No failure element. The only traces of the failed attempt are the attachment line for its error context and the duration: 2.306 seconds is the 2-second failed attempt plus the 0.3-second pass. A duration dashboard built from this file would call the coupon test slow, when its passing run took 0.3 seconds.
\n\nLoading the three files into the viewer, one run each, and copying its Markdown summary gave these lines:
\n\n| Run | Files | Tests | Passed | Failed | Errors | Skipped | Passed on retry | Time |\n| Run 1 | 1 | 3 | 2 | 1 | 0 | 0 | 0 | 2.31 s |\n| Run 2 | 1 | 3 | 1 | 2 | 0 | 0 | 0 | 4.03 s |\n| Run 3 | 1 | 3 | 2 | 1 | 0 | 0 | 0 | 308 ms |\n\n## Flaky candidates\n- `P F P` checkout.spec.ts › checkout › applies a coupon: coupon status\n\n## Consistent failures\n- checkout.spec.ts › checkout › rejects an expired card: payment status for card expiring 01/24The comparison catches the flake because build 2 failed it outright. Build 1's hidden retry still reads as P. The Playwright 1.62 and 1.63 source accepts an includeRetries reporter option that writes Surefire-style <flakyFailure> elements, although the options table on the reporters page does not list it. Rerunning build 1 with includeRetries: true changed the testcase's time to 0.307, added a <flakyFailure message=\"coupon status\" type=\"expect.toBe\">, and the sequence became R F P, with run 1 counted under \"Passed on retry\". Because the option is undocumented, test it on each Playwright upgrade before a gate depends on it.
build1/TEST-*.xml and build2/TEST-*.xml correctly when both are selected together. When every file is already a complete build, \"One run per file\" is the unambiguous choice.R means the run itself recorded a failed attempt followed by a pass; P F patterns across runs are the ones a single report cannot show.node_modules and pytest frames; the full text and system-out sit under each entry.Parsing uses the browser's DOMParser, and the comparison rules are fixed. Nothing is uploaded. A malformed file is rejected with its line and column, and the other files still load.
Tests are matched by classname plus name. A parameterised name that includes a timestamp, a random port, or a generated ID is a different test in every build, so its failures never line up and it never appears as flaky. Playwright's includeProjectInTestName and Vitest's name templates change identity the same way when they differ between builds.
\"Fails in every run\" only means every run you loaded. Three builds taken during a sandbox outage will list environment failures there. A test missing from any run is left out of that list, and skipped runs count as neither pass nor fail.
\n\nDurations are not comparable across producers. Surefire reports a flaky test's last successful run, Playwright's default sums attempts, and pytest includes fixtures unless junit_duration_report = call. Rank slow tests within one toolchain.
Large reports are parsed in memory. Twenty thousand test cases parse in well under a second in tests, but a file of hundreds of megabytes is limited by the tab's memory, and files over 100 MB are refused.
\n\nDoes a green build mean no test failed? No. With Surefire reruns, Gradle's mergeReruns, or Playwright retries, a test that failed and then passed leaves the build green. Surefire at least prints a Flakes: count in its summary line, and failOnFlakeCount, available since 3.0.0-M6, can fail the build on it.
Should Gradle's mergeReruns be on? Only if something reads <flakyFailure>. With it off, the failed execution stays visible as its own testcase, which the viewer reads as R for that run; with it on, a CI server that ignores the element shows a clean pass.
How many builds are enough? Enough to include the conditions the failure depends on. A flake that needs a slow dependency will not appear in three builds on a quiet day; a longer window, or builds on different runners, is a better sample than more builds in the same hour.
\n\nDownload the JUnit XML artifact from the last five builds on your main branch and load them into the JUnit XML Viewer as five runs. Save the Markdown summary with the build numbers. For each flaky candidate, paste its failure message into Test Failure Triage before editing the test. If you run Playwright, the guide to failing CI on flaky Playwright tests without hiding evidence covers the gate; on the JVM, making JUnit failures useful in CI covers stable names and retry configuration, and debugging JUnit parallel-execution flakes covers the common cause once you have the list.
\n\ntestsuite root, the flakes attribute, and flakyFailure, flakyError, rerunFailure and rerunError with a stackTrace child.failOnFlakeCount since 3.0.0-M6.rerunFailingTestsCount, with every failing attempt recorded.testsuites aggregate, with no retry elements.mergeReruns.--junit-xml, junit_family defaulting to xunit2, and junit_duration_report.testsuites wrapper and the separate testcase opened for a call failure followed by a teardown error.includeRetries branch.