Model JVM Test Layers with Gradle Test Suites

A team agrees that integration tests end in IT and unit tests end in Test, and that ./gradlew test runs only the second kind. Someone adds the filter to build.gradle. Eighteen months later the command takes eleven minutes and needs Docker running.

Nothing broke. The filter was moved during a refactor, or a new module never got one, or a class was named OrderRepositoryTest because it tests OrderRepository and it happens to open a connection. A convention that lives in a string pattern cannot report that it has been violated, and under JUnit Platform it is not even a discovery mechanism. Gradle states that there "only includes and excludes are used to filter test classes."

The alternative is to make the layer a build object rather than a naming habit. That is what the jvm-test-suite plugin models, and the reason it works is not the DSL: it is that a suite carries its own compile classpath, so a dependency that only integration tests are allowed to use cannot be imported from a unit test at all. The rule stops being social and starts being a compile error.

What you get before you configure anything

The plugin ships with Gradle and "is applied automatically by the java plugin", so a plain Java project already has a testing extension and a suite named test. Declaring a second one is four lines:

testing {
    suites {
        test {
            useJUnitJupiter('6.1.3')
        }

        integrationTest(JvmTestSuite) {
            dependencies {
                implementation project()
            }
        }
    }
}

That creates a source set at src/integrationTest/java, the configurations integrationTestImplementation, integrationTestCompileOnly and integrationTestRuntimeOnly, and a Test task named integrationTest. One suite, one target, one task. The plugin is explicit that this is currently a 1:1:1 relationship and that "future iterations of the plugin will allow defining multiple targets based on other attributes, such as a particular JDK/JRE runtime".

Two conventions in that snippet are worth stating outright, because they are the reverse of what most people guess. The built-in test suite has no framework configured by default: "it is always necessary to explicitly configure the built-in test suite even if JUnit 4 is desired". Every suite you add afterwards does have one: they "will by convention work as if useJUnitJupiter() was called".

Use the suite's own useJUnitJupiter() rather than the Test task's useJUnitPlatform(), because they are not equivalent despite the similar names. The suite method "automatically apply[s] the relevant testing framework libraries to the compile and runtime classpaths of the suite targets" as well as configuring the task, and its overload takes a version string. The task method only configures the task, leaving you to declare junit-jupiter yourself.

The dependency cliff past the built-in suite

Here is the sentence that will cost you a build failure and then explain a design. Of the automatic dependency on production code: "by convention, only the built-in test suite will automatically have a dependency on the production code of the project."

So implementation project() is not boilerplate. Without it, src/integrationTest/java cannot see your own classes at all. And with it you get less than test gets, because a project dependency "provides access to the project's outputs as well as any dependencies declared on its api and compileOnlyApi configurations", not its implementation ones.

Concretely: a library declared implementation 'com.fasterxml.jackson.core:jackson-databind:…' in the production block is on the unit-test compile classpath and is not on the integration-test compile classpath. Both are correct. implementation means "internal", and the integration suite is a separate consumer that has to ask for what it needs:

testing {
    suites {
        integrationTest(JvmTestSuite) {
            dependencies {
                implementation project()
                implementation 'org.testcontainers:testcontainers-postgresql:2.0.5'
                implementation 'org.postgresql:postgresql:42.7.4'
                runtimeOnly 'ch.qos.logback:logback-classic:1.5.6'
            }
        }

        contractTest(JvmTestSuite) {
            dependencies {
                implementation project()
                implementation 'io.rest-assured:rest-assured:5.4.0'
            }
        }
    }
}

This is the property the naming convention never had. A unit test cannot accidentally import PostgreSQLContainer, because that artefact is not on its compile classpath, and the compiler says so before anyone runs a build. Note the block's scoping too: inside a suite's dependencies you write implementation, compileOnly, runtimeOnly and annotationProcessor without prefixing the suite name, because "the dependencies block within a test suite is already scoped for that test suite."

When several suites share a stack, factor it with configureEach for all of them or withType(JvmTestSuite).matching { … }.configureEach for a subset, rather than repeating the list. Shared assertion libraries belong there; the database driver does not.

Cadence is a task graph, not a comment

A suite that nobody runs is worse than no suite, and the plugin will not run it for you. Two wiring steps, both documented as conventions you must override:

testing {
    suites {
        integrationTest(JvmTestSuite) {
            targets {
                all {
                    // By convention a suite target has no relationship to any
                    // other Test task, so the slow suite may start first.
                    testTask.configure { shouldRunAfter(test) }
                }
            }
        }
    }
}

// By convention test suite targets are not associated with the check task.
tasks.named('check') {
    dependsOn(testing.suites.integrationTest)
}

Which suites belong on check is the decision worth making deliberately, because it is the same question as "what must be true before this merges". A workable split for a service with four layers:

The last column is the one that decides the row. A suite whose failures are somebody else's to fix does not belong in the gate that blocks this pull request, and putting it there is how teams learn to rerun red builds.

Before trusting new wiring, verify it without paying for it. Gradle's --test-dry-run "will simulate the execution of the tests without actually running them. This will still generate reports, allowing for inspection of what tests were selected." That is the fastest way to confirm a suite picked up the classes you expected, and it costs seconds:

./gradlew integrationTest --test-dry-run
./gradlew check --dry-run          # the task graph, without running anything

Keeping suite identity in the results

Separate tasks give you separate results directories: JUnit XML lands in build/test-results/$testTaskName with a file per class. The HTML report does not preserve that split: it "automatically includes the results from all your Test tasks", so "if you add a Test task for integration tests, the report will include the results of both the unit tests and the integration tests if both tasks are run." Read the HTML for a human summary and parse the XML when you need to know which layer failed.

Two XML options are worth changing on day one, because the defaults predate the tooling that consumes them:

tasks.withType(Test).configureEach {
    reports {
        junitXml {
            // Attach log output to the individual test case rather than the
            // class. Gradle's own recommendation, and off by default.
            outputPerTestCase = true
            // Report a retried pass as <flakyFailure> instead of hiding it.
            mergeReruns = true
        }
    }
}

Gradle's guidance on the first is unambiguous: "most modern tools that observe JUnit XML files support the 'output per test case' format… it is recommended to enable this option as it provides more useful reporting." The second changes how a retried test is recorded rather than whether it is retried, which keeps a flake visible to a CI server that understands the format.

For a multi-project build, suite identity is carried in the dependency graph rather than in a path. Every suite "creates an outgoing variant containing its test execution results" tagged with org.gradle.testsuite.name, and the test-report-aggregation plugin matches on exactly that attribute to build one report per suite name across subprojects. One detail decides whether you ever see those reports: Gradle "stops executing tasks when any task fails — including test failures", so aggregation runs need --continue.

./gradlew testAggregateTestReport integrationTestAggregateTestReport --continue

Each suite gets its own execution budget

The point of the separation is that the layers stop having to agree. A unit suite wants many small JVMs; an integration suite wants few, because each one is holding a container.

testing {
    suites {
        test {
            targets.all {
                testTask.configure {
                    maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
                }
            }
        }

        integrationTest(JvmTestSuite) {
            targets.all {
                testTask.configure {
                    maxParallelForks = 1
                    maxHeapSize = '1g'
                    // Jupiter's own threading is configured per suite too.
                    systemProperty 'junit.jupiter.execution.parallel.enabled', 'false'
                }
            }
        }
    }
}

Note that testTask is a TaskProvider<Test>, so configure keeps it lazy. You can also configure the resulting task by name outside the suites block, which the documentation says is fine, and that is often easier to read for one-off settings that have nothing to do with suite modelling, such as a debugging JVM argument.

What the split actually costs

Three costs, none fatal, all worth knowing before you propose this.

The plugin is incubating. Gradle labels it "an incubating API and is subject to change in a future release", and the java_testing chapter refers to it the same way. In practice the DSL has been stable across releases, but incubating means a minor upgrade may require an edit, so the Gradle version belongs in your wrapper and in your CI image rather than floating.

Dependency duplication is real. Each suite declares its own assertion library, its own logging backend, its own mocking framework. configureEach reduces the repetition; it does not remove the fact that four suites means four classpaths to keep aligned when a version moves.

And a suite is a coarser unit than a tag. If half your integration tests need Postgres and half need Kafka, a suite per dependency is heavy and a tag inside one suite is lighter. The line to hold is the one from part one of this series: split into a suite when the runtime contract differs: different dependencies, different environment, different owner. Use tags to slice within a suite when everything except the selection is the same.

Apply this now

Run ./gradlew test --test-dry-run and read the class list. Every class in it that needs a database, a broker, a browser or a network route is a class that is currently inside your fast feedback loop, and the count is usually higher than the team's estimate.

Then create one suite named integrationTest and move exactly those classes into src/integrationTest/java. The compile errors you get are the finding: each one is a dependency that unit tests were carrying to serve integration tests. Add them to the new suite's dependencies block and delete them from testImplementation.

Finish by wiring check to the new suite and adding shouldRunAfter(test), then compare ./gradlew test before and after. What you should be able to show is a shorter unit run, a testImplementation block with fewer entries in it, and a second results directory under build/test-results/ that did not exist yesterday.

Questions about modelling JVM test layers

Is this better than declaring a source set and a Test task by hand?

Yes, for one reason that outlives the syntax: the outgoing variant. A hand-rolled task produces results that nothing else can find by suite name, so cross-project aggregation has to be wired manually with attributes you invent. The suite publishes org.gradle.testsuite.name, and test-report-aggregation consumes it. Everything else the plugin does (source set, configurations, task) you could write yourself in about fifteen lines.

Where should shared test helpers live?

In a place both suites depend on, which usually means Gradle's test fixtures support or a separate module. What does not work is integrationTestImplementation sourceSets.test.output, which reintroduces exactly the coupling the split existed to remove: the integration suite would then compile against the unit suite's dependencies, and a unit test could once again import a container library transitively.

How many suites is too many?

The count that matters is how many distinct environments your tests need, and for most services that is two or three. If two suites have the same dependencies, the same runtime requirements and the same owner, they are one suite with two tags. Creating smokeTest as a fifth suite when it runs in the same environment as acceptanceTest buys a task name and costs a classpath.

Do the build cache and up-to-date checks still work?

Each suite's Test task is an ordinary Gradle task with its own inputs, so it is skipped when nothing it depends on changed. That is most of the benefit of the split on a developer machine. The caveat is the same one that applies to any test task: if a test reads something Gradle does not know about, such as a container image tag from the environment, Gradle cannot see that input changed and will happily report the task up to date. Declare those as inputs explicitly or accept that the suite needs --rerun-tasks occasionally.

Can one suite use a different testing framework from another?

Yes, and it is one of the cleaner migration paths off an old framework. useJUnit(), useJUnitJupiter(), useTestNG() and their versioned overloads are per suite, so a legacy suite can stay on JUnit 4 while new work goes into a Jupiter suite, with both wired to check. The framework choice then shrinks over time by moving classes between source directories rather than by a flag day.

Primary references

• The JVM Test Suite plugin: Gradle 9.7.1 user manual. That the plugin "is an incubating API and is subject to change in a future release" and "is applied automatically by the java plugin"; the objects it adds, including the test suite, its source set and the testImplementation/testCompileOnly/testRuntimeOnly configurations; the current 1:1:1 relationship between suite, target and Test task; that "it is always necessary to explicitly configure the built-in test suite even if JUnit 4 is desired" while other suites "by convention work as if useJUnitJupiter() was called"; that "by convention, only the built-in test suite will automatically have a dependency on the production code of the project" and that implementation project() "provides access to the project's outputs as well as any dependencies declared on its api and compileOnlyApi configurations"; that a suite's dependencies block "is already scoped for that test suite"; that "by convention, test suite targets have no relationship to other Test tasks" and "are not associated with the check task", with the shouldRunAfter and dependsOn examples; the conventional src/integrationTest/java location and how to override it; the configureEach and withType(...).matching(...).configureEach sharing patterns; that testTask is a TaskProvider<Test> and tasks may also be configured by name; the difference between JvmTestSuite's framework methods and the Test task's, since the former also apply the framework libraries to the suite's classpaths; and the outgoing variant carrying org.gradle.testsuite.name.
• Gradle 9.7.1: test reporting, JUnit XML options and dry runs. That the HTML report "automatically includes the results from all your Test tasks", so a report covers unit and integration results together; that JUnit XML is written to build/test-results/$testTaskName with a file per class; the outputPerTestCase option, off by default, with the recommendation to enable it, and mergeReruns recording a retried pass as <flakyFailure>; maxParallelForks defaulting to 1 and forkEvery defaulting to 0; dryRun and --test-dry-run, which "will still generate reports, allowing for inspection of what tests were selected"; and that "with JUnit Platform, only includes and excludes are used to filter test classes". Gradle 9.7.1.
• The Test Report Aggregation plugin: Gradle 9.7.1 user manual. That the plugin "takes no action unless applied in concert with the JVM Test Suite Plugin"; that it adds a testSuiteAggregateTestReport task per suite which matches variants on org.gradle.category = verification, org.gradle.testsuite.name and org.gradle.verificationtype = test-results; the testReportAggregation and aggregateTestReportResults configurations; and that because Gradle "stops executing tasks when any task fails — including test failures", generating aggregation reports requires --continue.
• JUnit 6.1.3: supported Java versions and module layout. The current JUnit version used in the useJUnitJupiter('6.1.3') example, and that JUnit requires Java 17 or higher at runtime, a constraint that belongs in the suite's toolchain decision.
• Testcontainers 2.0.5: the Postgres module artefact. The org.testcontainers:testcontainers-postgresql:2.0.5 coordinate used in the integration suite's dependency block, and the reminder that "adding this Testcontainers library JAR will not automatically add a database driver JAR to your project", which is why the JDBC driver is declared beside it.