Skip to main content
automationtester.in
Back to Blog
Tools & Reviews

Find Flaky Tests by Comparing JUnit XML Runs

A single JUnit report can record a failed-then-passed test as a plain pass. Three real Playwright builds show what each producer writes for a retry and how comparing runs exposes the flake.

Shashank Rawlani

Engineering Leader | Builder | Problem Solver | AI Engineer

LinkedIn Portfolio
Published Oct 23, 20268 min read
View AI-ready Markdown
Three report sheets with matching rows; one row reads green, orange, green and is pulled out to a ringed green node, while a row that is orange in every sheet flows into its own orange panel.
Listen to this article · 11 min
Load the JUnit XML report from each of several CI builds into the JUnit XML Viewer, one run per build. It lists tests that passed in one build and failed in another, with a pass/fail sequence per test, and keeps tests that failed in every build in a separate list. Read the retry elements too: a single report can record a test that failed and then passed as an ordinary pass.

A Playwright build finishes with one red test, rejects an expired card. The team fixes it and moves on. The same report also contains applies a coupon, which failed its first attempt, passed on retry, and appears in the XML as a passing <testcase> with no <failure> child. Nothing in that file says it was ever red. The only way to see it is to compare it with other builds, or to make the reporter keep the failed attempt.

How each producer writes a retried test

There is no single JUnit XML standard. The Ant JUnit task made the format common; Maven Surefire publishes its own XSD; the rest follow the Ant shape loosely. What differs most, and matters most for flakes, is what happens to a failed attempt that is followed by a pass.

ProducerFailed attempt, then a passRecorded time
Maven Surefire and Failsafe with rerunFailingTestsCountOne <testcase> with a <flakyFailure> or <flakyError> per failed attempt; the XSD also defines a flakes count on the suiteThe last successful run
Maven Surefire, every rerun fails<failure> for the first run plus one <rerunFailure> per rerunThe first failing run
Gradle, defaultEach execution is a separate <testcase> with the same namePer execution
Gradle with mergeReruns = trueOne <testcase> with <flakyFailure>, the Surefire shapeNot stated in the manual
Playwright junit reporter, defaultA plain passing <testcase>The sum of every attempt

Two details in that table cause most misreadings. In Surefire's schema, a rerun's trace sits inside a <stackTrace> child element, while the first failure's trace is the <failure> element's own text, so a parser that only reads element text finds empty reruns. And Gradle warns that a CI server that does not understand <flakyFailure> "will indicate that the test succeeded". The manual names Jenkins' JUnit plugin, which leaves those failed executions out of its test result views.

Other producers differ in ways that affect matching rather than retries. pytest wraps a single <testsuite> in a <testsuites> root and, when a test fails in its call and then errors in teardown, writes two <testcase> elements for it. Its junit_duration_report setting defaults to total, so a test's time includes setup and teardown. Vitest names each case with its ancestor describe blocks joined by > ; jest-junit's default titleTemplate is {classname} {title}. Change either template between builds and the same test gets a new identity.

Three real Playwright builds, compared

The reports below came from Playwright 1.62.1 with retries: 1. A BUILD variable stands in for a coupon service that is sometimes slow: build 1 fails the coupon test's first attempt only, build 2 fails both attempts, and build 3 never fails. The expired-card test is a real bug and fails every time.

// playwright.config.ts
export default defineConfig({
  retries: 1,
  reporter: [["junit", { outputFile: `results/build-${process.env.BUILD}.xml` }]],
})

// tests/checkout.spec.ts
test("applies a coupon", async ({}, testInfo) => {
  const fails = (build === 1 && testInfo.retry === 0) || build === 2
  await new Promise(resolve => setTimeout(resolve, fails ? 2000 : 300))
  expect(fails ? "timeout" : "applied", "coupon status").toBe("applied")
})
for b in 1 2 3; do BUILD=$b npx playwright test; done

Build 1's report records the flaky test like this:

<testcase name="checkout › applies a coupon" classname="checkout.spec.ts" time="2.306">
<system-out>
<![CDATA[
[[ATTACHMENT|../test-results/checkout-checkout-applies-a-coupon/error-context.md]]
]]>
</system-out>
</testcase>

No failure element. The only traces of the failed attempt are the attachment line for its error context and the duration: 2.306 seconds is the 2-second failed attempt plus the 0.3-second pass. A duration dashboard built from this file would call the coupon test slow, when its passing run took 0.3 seconds.

Loading the three files into the viewer, one run each, and copying its Markdown summary gave these lines:

| Run | Files | Tests | Passed | Failed | Errors | Skipped | Passed on retry | Time |
| Run 1 | 1 | 3 | 2 | 1 | 0 | 0 | 0 | 2.31 s |
| Run 2 | 1 | 3 | 1 | 2 | 0 | 0 | 0 | 4.03 s |
| Run 3 | 1 | 3 | 2 | 1 | 0 | 0 | 0 | 308 ms |

## Flaky candidates
- `P F P` checkout.spec.ts › checkout › applies a coupon: coupon status

## Consistent failures
- checkout.spec.ts › checkout › rejects an expired card: payment status for card expiring 01/24

The comparison catches the flake because build 2 failed it outright. Build 1's hidden retry still reads as P. The Playwright 1.62 and 1.63 source accepts an includeRetries reporter option that writes Surefire-style <flakyFailure> elements, although the options table on the reporters page does not list it. Rerunning build 1 with includeRetries: true changed the testcase's time to 0.307, added a <flakyFailure message="coupon status" type="expect.toBe">, and the sequence became R F P, with run 1 counted under "Passed on retry". Because the option is undocumented, test it on each Playwright upgrade before a gate depends on it.

Reading a set of reports in the viewer

  1. Choose how a selection of files maps to runs. Surefire and Gradle write one file per test class, so a build is many files. The default grouping puts files that share no test ID into one run and starts a new run when a file repeats IDs. That splits build1/TEST-*.xml and build2/TEST-*.xml correctly when both are selected together. When every file is already a complete build, "One run per file" is the unambiguous choice.
  2. Select the files, drop them on the page, or paste one report. Each addition becomes one or more runs, in the order added. Order matters, because the sequence reads oldest to newest.
  3. Read "Flaky candidates" first. R means the run itself recorded a failed attempt followed by a pass; P F patterns across runs are the ones a single report cannot show.
  4. Treat "Fails in every run" as a bug list, not a flake list. Retrying a test that fails every time only delays the red build.
  5. Use "Failures and errors" for the messages. Traces are trimmed by hiding JUnit, Surefire, Gradle, reflection, node_modules and pytest frames; the full text and system-out sit under each entry.
  6. Copy the Markdown summary into the pull request or issue, or download the JSON for a script.

Parsing uses the browser's DOMParser, and the comparison rules are fixed. Nothing is uploaded. A malformed file is rejected with its line and column, and the other files still load.

Where the comparison misleads

Tests are matched by classname plus name. A parameterised name that includes a timestamp, a random port, or a generated ID is a different test in every build, so its failures never line up and it never appears as flaky. Playwright's includeProjectInTestName and Vitest's name templates change identity the same way when they differ between builds.

"Fails in every run" only means every run you loaded. Three builds taken during a sandbox outage will list environment failures there. A test missing from any run is left out of that list, and skipped runs count as neither pass nor fail.

Durations are not comparable across producers. Surefire reports a flaky test's last successful run, Playwright's default sums attempts, and pytest includes fixtures unless junit_duration_report = call. Rank slow tests within one toolchain.

Large reports are parsed in memory. Twenty thousand test cases parse in well under a second in tests, but a file of hundreds of megabytes is limited by the tab's memory, and files over 100 MB are refused.

Questions about comparing JUnit reports

Does a green build mean no test failed? No. With Surefire reruns, Gradle's mergeReruns, or Playwright retries, a test that failed and then passed leaves the build green. Surefire at least prints a Flakes: count in its summary line, and failOnFlakeCount, available since 3.0.0-M6, can fail the build on it.

Should Gradle's mergeReruns be on? Only if something reads <flakyFailure>. With it off, the failed execution stays visible as its own testcase, which the viewer reads as R for that run; with it on, a CI server that ignores the element shows a clean pass.

How many builds are enough? Enough to include the conditions the failure depends on. A flake that needs a slow dependency will not appear in three builds on a quiet day; a longer window, or builds on different runners, is a better sample than more builds in the same hour.

Compare this week's builds

Download the JUnit XML artifact from the last five builds on your main branch and load them into the JUnit XML Viewer as five runs. Save the Markdown summary with the build numbers. For each flaky candidate, paste its failure message into Test Failure Triage before editing the test. If you run Playwright, the guide to failing CI on flaky Playwright tests without hiding evidence covers the gate; on the JVM, making JUnit failures useful in CI covers stable names and retry configuration, and debugging JUnit parallel-execution flakes covers the common cause once you have the list.

JUnit XML format references