exceptionFormat = 'full' (the default console drops your assertion message), build names from fixed case slugs, attach sanitized evidence with TestReporter (captured into reports since Gradle 9.4), and make sure something reads the flaky outcome of a retry.Here is a failed contract test as it appears in a CI console, in full:
\n\nOrderContractTest > contract(Order) > [2] order = Order[id=ord_7f3a9c21, createdAt=2026-09-30T08:14:22.391Z, lines=3] FAILED\n org.opentest4j.AssertionFailedError at OrderContractTest.java:47The identity contains a generated id and a timestamp, so this failure has no history: yesterday's run of the same case had a different name, and a dashboard keyed on names sees a new test that failed once. There is no endpoint, no field, no expected value, and not even the assertion message, although the test may well have had one.
\n\nThe fix is a handful of decisions made once, in the build and in a shared helper, so individual tests get them for free. The worked example is a parameterized contract test for GET /orders/{id}, where one case, an order stored before currencies were recorded, comes back without $.total.currency.
org.gradle.test-retry 1.6.6; Maven Surefire 3.6.0. Two behaviours below are recent enough to depend on your versions: file and directory attachments on TestReporter arrived in Jupiter 5.12, and Gradle only started carrying TestReporter data into its HTML and XML reports in 9.4.0.Gradle logs test events per log level, and CI runs at LIFECYCLE. At that level the documented default is \"Test failures\", with \"Exception format is SHORT\". The Javadoc for TestExceptionFormat.SHORT is precise about what that leaves out: it \"shows the exception types and locations as well as the hierarchy of causes, but not exception messages or full stack traces.\"
So the assertion message never reaches the console by default. The XML and the HTML report under build/reports/tests have it, but the first thing anyone reads when a pull request goes red is the job log, which says AssertionFailedError at OrderContractTest.java:47. FULL is only the default at INFO and DEBUG, and nobody runs CI at those levels.
Set it once, for every test task:
\n\ntasks.withType(Test).configureEach {\n testLogging {\n events 'failed', 'skipped'\n exceptionFormat = 'full'\n // Deliberately not showStandardStreams: see the section on stdout.\n }\n reports.junitXml {\n outputPerTestCase = true\n mergeReruns = true\n }\n}The cost is log volume on a bad day: a hundred failures print a hundred messages and stack traces. That is the right trade against a hundred people opening a report to read a message that could have been in the log.
\n\noutputPerTestCase \"associates any output logging generated during a test case to that test case in the results\"; with it off, the default, output belongs to the class as a whole, so the lines printed by the failing invocation sit beside the lines from every passing one. mergeReruns is less harmless than it looks; it gets its own section below.
The reported name is a test's key everywhere downstream: CI history, flaky-test dashboards, \"failing since\" columns, and Gradle's own rerun handling. Gradle's documentation for mergeReruns says so directly: \"Tests are grouped and merged based on their reported name\", and it asks that names from parameterization or \"any other kind of mechanism that produces a potentially dynamic test name\" be kept stable.
The opening example breaks that rule without anyone deciding to. Its argument source builds orders with a random id and Instant.now(), and the default display name prints the index and each argument's string form. Order is a record, and a record's implicitly declared toString() is, in the Java SE API's words, \"derived from all of the component fields\", so the identity of the test is now partly a clock reading.
// Wrong: the display name is derived from generated data.\nstatic Stream<Order> orders() {\n return Stream.of(\n Order.random(),\n Order.legacy(Instant.now()), // prints createdAt into the test name\n Order.withLines(3));\n}\n\n@ParameterizedTest\n@MethodSource(\"orders\")\nvoid contract(Order order) {\n var response = client.get(\"/orders/\" + order.id());\n assertTrue(response.body().contains(\"\\\"currency\\\"\"));\n}Part two of this series covered naming rows with argumentSet so that a reorder does not change what a name points at. CI needs a stricter version of that rule: build the name only from things written in the source file. A case slug qualifies. A generated id, a timestamp, a port, a container hostname or a random seed never does, however useful while debugging. Those values belong in the attached evidence, not in the key, where they fragment history.
static Stream<Arguments> storedOrders() {\n return Stream.of(\n argumentSet(\"complete-order\", StoredOrder.complete(\"ord_complete\")),\n argumentSet(\"multi-line-order\", StoredOrder.withLines(\"ord_three_lines\", 3)),\n argumentSet(\"missing-currency\", StoredOrder.legacyWithoutCurrency(\"ord_legacy_no_ccy\")));\n}\n\n@ParameterizedTest(name = \"GET /orders/'{id}' [{argumentSetName}]\")\n@MethodSource(\"storedOrders\")\nvoid honoursTheOrderContract(StoredOrder stored, TestReporter reporter) throws Exception {\n // body in the next two sections\n}One detail in that name attribute will bite anyone who copies an endpoint template into it. The JUnit guide states that the attribute \"is a MessageFormat pattern\", and in MessageFormat a brace pair is a format element whose index must be digits. To print a literal {id}, quote it: the MessageFormat Javadoc gives the rule as \"pattern string \"'{0}'\" represents string \"{0}\", not a FormatElement\". The rendered name is then GET /orders/{id} [missing-currency], which is exactly what someone will paste into a search box.
The failure message has one job: say which part of the contract broke, what it requires, and what came back. Aim for a record a reviewer can read without the code:
\n\ntest: GET /orders/{id} [missing-currency]\npath: $.total.currency\nexpected: required string\nactual: absent\nattachment: order-missing-currency.json (sanitized response)Build that once as a helper, rather than trusting each test author to format it:
\n\nimport com.fasterxml.jackson.databind.JsonNode; // Jackson 2.x\n\nfinal class Contract {\n\n /** Fails with \"path: expected required string, actual …\". */\n static void requiredString(JsonNode body, String jsonPath, String pointer) {\n JsonNode value = body.at(pointer);\n assertTrue(value.isTextual(), () -> jsonPath\n + \": expected required string, actual \"\n + describe(value));\n }\n\n private static String describe(JsonNode value) {\n if (value.isMissingNode()) return \"absent\";\n if (value.isNull()) return \"null\";\n return value.getNodeType().name().toLowerCase() + \" \" + value;\n }\n}The message is a Supplier<String> because Jupiter documents that a supplier \"is only evaluated when the assertion fails\", so the formatting work is paid for only on failures.
A wrong response is often wrong in more than one place, and stopping at the first failed assertion hides the second until the next CI run. In assertAll, \"all assertions are executed, and all failures will be reported together\" under a heading you supply. Use the test identity as the heading.
JsonNode body = json.readTree(response.body());\nassertAll(\"GET /orders/{id} [\" + stored.slug() + \"]\",\n () -> assertEquals(200, response.statusCode(), \"HTTP status\"),\n () -> Contract.requiredString(body, \"$.total.currency\", \"/total/currency\"),\n () -> Contract.requiredString(body, \"$.total.amount\", \"/total/amount\"),\n () -> Contract.requiredString(body, \"$.status\", \"/status\"));The path is written twice, as a reader expects it and as the JSON Pointer Jackson's at() takes. If the expected shape lives in a JSON Schema, the JSON Schema generator is a quick check that the fields your helper calls required match the schema's.
One caution: describe() prints the actual value, and the message goes verbatim into the XML, the console and every CI log archive. For a currency code that is fine. For a field that can hold an email or a token, print type and length only.
The message says what broke; the response body says why. The wrong place for that body is standard output. The right place is TestReporter, which the user guide describes as the Jupiter replacement for printing: \"you should use TestReporter where you used to print information to stdout or stderr\".
publishEntry for key-value data has been there since Jupiter 5.0. publishFile and publishDirectory, which attach files to the current test, arrived in 5.12. In 5.14 the file overloads moved to org.junit.jupiter.api.MediaType; the ones taking org.junit.jupiter.api.extension.MediaType are deprecated for removal, so check that import after an upgrade.
@ParameterizedTest(name = \"GET /orders/'{id}' [{argumentSetName}]\")\n@MethodSource(\"storedOrders\")\nvoid honoursTheOrderContract(StoredOrder stored, TestReporter reporter) throws Exception {\n orders.seed(stored);\n HttpResponse<String> response = client.get(\"/orders/\" + stored.id());\n\n reporter.publishEntry(Map.of(\n \"endpoint\", \"GET /orders/{id}\",\n \"case\", stored.slug(),\n \"orderId\", stored.id(),\n \"status\", String.valueOf(response.statusCode()),\n \"requestId\", response.headers().firstValue(\"x-request-id\").orElse(\"none\")));\n\n try {\n JsonNode body = json.readTree(response.body());\n assertAll(\"GET /orders/{id} [\" + stored.slug() + \"]\",\n () -> assertEquals(200, response.statusCode(), \"HTTP status\"),\n () -> Contract.requiredString(body, \"$.total.currency\", \"/total/currency\"),\n () -> Contract.requiredString(body, \"$.total.amount\", \"/total/amount\"));\n } catch (AssertionError failure) {\n // Evidence only for failures, so passing runs do not fill the artefact store.\n reporter.publishFile(\"order-\" + stored.slug() + \".json\", MediaType.APPLICATION_JSON,\n file -> Files.writeString(file, Redactor.redact(response.body())));\n throw failure;\n }\n}The request id goes into an entry because it is the join key to server logs, and exactly the per-run value that must stay out of the name. The file is published only on failure, so the artefact store grows with problems rather than with the suite. The price is that a green run leaves nothing to compare against; for the few tests where \"what did it look like when it passed\" matters, publish unconditionally.
\n\nFor years, Jupiter report entries could show up in an IDE but not in Gradle's reports. Gradle 9.4.0, released 2026-03-04, changed that: it \"now captures this metadata and integrates it directly into both the HTML test report and the XML test results\". The HTML report gains a Data tab for key-value entries and an Attachments tab linking to published files, and the release notes say some known media types, \"such as images and videos, are rendered directly\". In the JUnit XML, entries are \"mapped to <properties/>\" and each file is written in the form [[ATTACHMENT|/path/to/file]], a convention those notes attribute to Jenkins, Azure Pipelines and GitLab.
On an older Gradle the same code runs and the data goes nowhere a CI server can see. The alternative JUnit documents is the Open Test Reporting listener in junit-platform-reporting, off until junit.platform.reporting.open.xml.enabled=true, which writes one XML file that \"supports all features of the JUnit Platform such as hierarchical test structures, display names, tags, etc.\" It helps only if your CI server reads that format.
Put together, the console line and the report tabs for the failing case read like this (the console lines are what exceptionFormat = 'full' prints; the last two rows are the HTML report's Data and Attachments tabs):
OrderContractTest > honoursTheOrderContract(StoredOrder, TestReporter) > GET /orders/{id} [missing-currency] FAILED\n org.opentest4j.MultipleFailuresError: GET /orders/{id} [missing-currency] (1 failure)\n org.opentest4j.AssertionFailedError: $.total.currency: expected required string, actual absent\n\n Data endpoint=GET /orders/{id} case=missing-currency orderId=ord_legacy_no_ccy\n status=200 requestId=5c1e0b7a\n Attachments order-missing-currency.jsonNothing in the name changes between runs. Everything that does is in the data, attached to the invocation that produced it.
\n\nGradle's JUnit XML includes standard output and standard error by default (includeSystemOutLog and includeSystemErrLog both default to true), so a println of a response in a test that authenticates will land in the result files, and from there in every CI artefact store and every tool that ingests them. showStandardStreams goes one step further and puts it in the job log. That is why the configuration above leaves it off.
Richer formats carry richer leaks. Open Test Reporting can include Git information, which its README says to include \"only if the requesting user has opted in explicitly\" because it \"may include file names that contain personally identifiable or otherwise sensitive information\". JUnit's junit.platform.reporting.open.xml.git.enabled defaults to false; leave it there.
Put redaction in one place, applied to every attachment before it is written, and never attach request headers at all, since that is where the Authorization value lives:
final class Redactor {\n\n private static final ObjectMapper MAPPER = new ObjectMapper();\n private static final Pattern SENSITIVE =\n Pattern.compile(\"(?i).*(token|secret|password|authorization|email|card).*\");\n\n static String redact(String json) throws IOException {\n JsonNode tree = MAPPER.readTree(json);\n scrub(tree);\n return MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(tree);\n }\n\n private static void scrub(JsonNode node) {\n if (node instanceof ObjectNode object) {\n List<String> keys = new ArrayList<>();\n object.fieldNames().forEachRemaining(keys::add);\n for (String key : keys) {\n if (SENSITIVE.matcher(key).matches()) {\n object.put(key, \"[redacted]\");\n } else {\n scrub(object.get(key));\n }\n }\n } else if (node instanceof ArrayNode array) {\n array.forEach(Redactor::scrub);\n }\n }\n}Redacting by key keeps the structure, which is the evidence here: the attachment still shows total with an amount and no currency. The weakness is that a denylist knows only the names it was given, so a new apiKey field passes through. If payloads often carry credentials, invert it into an allowlist of keys whose values may appear.
Retries are where reports most often stop telling the truth, through two defaults that are each reasonable alone. The Gradle Test Retry plugin reruns failed tests after the rest of the task, and its failOnPassedAfterRetry setting \"defaults to false, which results in the task not failing if all tests pass on retry\". Without mergeReruns, the plugin's README states that \"each execution of a test is discretely reported in Gradle-generated XML and HTML reports\", so a CI server reading the XML may still count the failed one.
Turn on mergeReruns and the failures are recorded \"as <flakyFailure> instead of <failure>, within one <testcase>\". Gradle then states the consequence: if your CI server does not understand that element, \"it will indicate that the test succeeded\". Maven Surefire writes the same shape when rerunFailingTestsCount is set, and records that \"the running time of a flaky test will be the running time of the last successful run\".
So a test that failed on its first attempt yields a green build, a passing testcase and a duration from its good run. The failure is in the file, in an element nobody downstream reads.
\n\nThe README lists Jupiter 5.9.2 as the version it was tested against, noting that other versions \"are likely to work\", so push one deliberately failing test through it on 6.x before relying on it. For a Jupiter @ParameterizedTest \"the plugin will retry the entire method with all parameters including those that initially passed\". Retrying [missing-currency] reruns every case, and any name that differs between the first execution and the retry cannot be merged at all.
Pick one of two honest configurations. The first keeps retries as a diagnostic and makes flakiness fail the build:
\n\nplugins {\n id 'org.gradle.test-retry' version '1.6.6'\n}\n\ntasks.withType(Test).configureEach {\n retry {\n maxRetries = 2\n maxFailures = 10 // stop retrying when the database is down, not flaky\n failOnPassedAfterRetry = true\n }\n}The build is red either way, but the report now separates \"fails every time\" from \"failed, then passed\": look at the code, or look for a race. maxFailures covers the README's case for not retrying at all, \"if a disk fills up or a required database is not available\". On Maven, failOnFlakeCount, available since Surefire 3.0.0-M6, fails the build once more than a given number of tests are flaky.
The second accepts green-on-retry but refuses to let the flake vanish, by counting it straight from the XML after the test task:
\n\n#!/usr/bin/env bash\n# Run after ./gradlew test, whatever its exit code.\nflaky=$(grep -l '<flakyFailure' build/test-results/test/*.xml 2>/dev/null | wc -l | tr -d ' ')\nif [ \"$flaky\" -gt 0 ]; then\n echo \"::warning::$flaky result file(s) contain tests that passed only on retry\"\n grep -h -o 'testcase name=\"[^\"]*\"' $(grep -l '<flakyFailure' build/test-results/test/*.xml)\nfiCrude, and enough: an element nobody reads becomes a job-summary line naming the tests. It works only because the names are stable. A flaky test whose name carries a timestamp is a different test every week, and nobody acts on a list of names that never repeat. To see those names across several builds rather than one, load each build's XML into the JUnit XML Viewer, which reads <flakyFailure> and prints every test's pass and fail sequence by build.
Take your most recent failed CI run and, without opening the source or rerunning anything, answer four questions from what the job left behind: which case, which field, expected against actual, and whether it failed on the first attempt. The ones you could not answer are your backlog, in order.
\n\nIf the message is missing from the log, add exceptionFormat = 'full' today. Then grep a month of result files for names containing a timestamp or anything shaped like a UUID. Each hit is a test with no history; the fix is a case slug.
Last, check whether retries are enabled anywhere, and if so, find out what reads <flakyFailure>. If nothing does, set failOnPassedAfterRetry = true or add the counting step in the same change. Keep one before-and-after pair of console lines for a real failure in the pull request, so whoever wants to revert a setting can see what it bought.
It writes the legacy format: JUnit's LegacyXmlReportGeneratingListener, which the guide describes as \"compatible with the de facto standard for JUnit 4 based test reports that was made popular by the Ant build system\", producing a separate file per root in the test plan when you pass --reports-dir. Gradle writes its own XML in the same Ant-derived format, with options such as mergeReruns that the launcher does not have.
Because JUnit's own capture, enabled with junit.platform.output.capture.stdout, records only \"output emitted by the thread that was used to execute a container or test\". Anything an HTTP client or an executor prints from its own threads is omitted, since it cannot be attributed to a test. Publish evidence through TestReporter from the test thread instead.
It can cluster them usefully, provided the input is the structured result files after redaction rather than raw logs, and provided every cluster it reports links back to the exact test names it grouped. A summary that cannot be traced to test identities cannot be checked, and one built from unredacted logs has sent your tokens somewhere new.
\n\nmergeReruns still worth enabling?Yes, if something reads <flakyFailure>: a CI server that understands it, the counting step above, or Develocity, which Gradle notes detects flaky tests \"regardless of this setting\". Without one of those, merging makes a flake look like an ordinary pass, and leaving it off at least keeps the failed execution visible as its own testcase.
LIFECYCLE, \"Test failures\" with \"Exception format is SHORT\"); the JUnit XML location test-results/$testTaskName with a file per class; includeSystemOutLog and includeSystemErrLog defaulting to true; outputPerTestCase, off by default, which \"associates any output logging generated during a test case to that test case\"; and mergeReruns, which records failures \"as <flakyFailure> instead of <failure>\", with the statements that a CI server that does not understand this \"will indicate that the test succeeded\" and that \"tests are grouped and merged based on their reported name\".SHORT \"shows the exception types and locations as well as the hierarchy of causes, but not exception messages or full stack traces\".exceptionFormat defaults to FULL for INFO and DEBUG and SHORT for LIFECYCLE; showStandardStreams as the switch for logging standard out and error; stackTraceFilters; and the events property.reporting.baseDirectory.dir(\"tests\"), e.g. build/reports/tests, and test result XML to build/test-results.TestReporter entries and files, \"into both the HTML test report and the XML test results\"; the Data and Attachments tabs; direct rendering of some media types; entries \"mapped to <properties/>\" and files written as [[ATTACHMENT|/path/to/file]]. Released 2026-03-04.retry extension; that failOnPassedAfterRetry defaults to false so a pass on retry does not fail the task; maxFailures and its disk-full and missing-database rationale; that for a Jupiter @ParameterizedTest it \"will retry the entire method with all parameters including those that initially passed\"; and that \"each execution of a test is discretely reported in Gradle-generated XML and HTML reports\". Version 1.6.6 per the Gradle Plugin Portal listing.rerunFailingTestsCount; the flakyFailure, flakyError and rerunFailure elements; that a flaky test's recorded time is \"the running time of the last successful run\"; and failOnFlakeCount since 3.0.0-M6.publishEntry, publishFile and publishDirectory with examples, and the advice to \"use TestReporter where you used to print information to stdout or stderr\".org.junit.jupiter.api.MediaType overloads since 5.14 with the extension.MediaType variants deprecated for removal; and that a published name must not contain path separators.name \"is a MessageFormat pattern\"; the placeholders including {argumentSetName}; junit.jupiter.params.displayname.default and the precedence of the annotation attribute over it; and the default name built from the index and each argument's string form.\"'{0}'\" represents string \"{0}\", not a FormatElement\", and that an argument index is written with digits.Supplier<String> message is \"only evaluated when the assertion fails\", and that in assertAll \"all assertions are executed, and all failures will be reported together\".junit.platform.reporting.open.xml.enabled and git.enabled parameters, the Gradle wiring, and the legacy listener's Ant-compatible format and its use by the Console Launcher.--reports-dir option that enables XML report output.toString() is \"derived from all of the component fields\", which is how a generated id and a timestamp end up in a default parameterized display name.