Review AI-Generated JUnit Tests Before They Raise Coverage

The ticket, PRICE-212, is two sentences long: orders of 10 or more units take 5% off the line, and the line total is rounded HALF_UP to two decimal places. The implementation is small enough to read in one breath.

public final class BulkDiscount {

    static final int THRESHOLD = 10;
    private static final BigDecimal MULTIPLIER = new BigDecimal("0.95");

    public BigDecimal apply(BigDecimal gross, int quantity) {
        BigDecimal net = quantity >= THRESHOLD ? gross.multiply(MULTIPLIER) : gross;
        return net.setScale(2, RoundingMode.HALF_UP);
    }
}

public final class Checkout {

    private final Catalogue catalogue;
    private final BulkDiscount discount;

    // constructor omitted

    public BigDecimal lineTotal(String sku, int quantity) {
        BigDecimal gross = catalogue.unitPrice(sku).multiply(BigDecimal.valueOf(quantity));
        return discount.apply(gross, quantity);
    }
}

An assistant was asked for tests and returned these. Both classes compile, both pass, and the package reports every line and every branch covered.

class BulkDiscountTest {

    private final BulkDiscount discount = new BulkDiscount();

    @Test
    void discountsLargeOrders() {
        BigDecimal gross = new BigDecimal("100.00");
        BigDecimal expected = gross.multiply(new BigDecimal("0.95")).setScale(2, RoundingMode.HALF_UP);
        assertEquals(expected, discount.apply(gross, 50));
    }

    @Test
    void leavesSmallOrdersAlone() {
        assertEquals(new BigDecimal("20.00"), discount.apply(new BigDecimal("20.00"), 2));
    }
}

class CheckoutTest {

    @Test
    void appliesBulkDiscountToLargeOrders() {
        Catalogue catalogue = mock(Catalogue.class);
        BulkDiscount discount = mock(BulkDiscount.class);
        when(catalogue.unitPrice("SKU-1")).thenReturn(new BigDecimal("2.00"));
        when(discount.apply(any(), anyInt())).thenReturn(new BigDecimal("95.00"));

        assertEquals(new BigDecimal("95.00"), new Checkout(catalogue, discount).lineTotal("SKU-1", 50));
    }
}

Now break the ticket three ways, one at a time: set THRESHOLD to 11, swap HALF_UP for DOWN, swap it for HALF_EVEN. Every build stays green. Those are the three most plausible ways to misread PRICE-212, and the suite notices none of them.

Identical JaCoCo rows for a useless suite and a useful one

Here is the JaCoCo CSV for the package after the generated tests ran, trimmed to the columns that matter. After the review, which replaced every one of those tests, the untrimmed file is identical down to the byte.

CLASS,INSTRUCTION_MISSED,INSTRUCTION_COVERED,BRANCH_MISSED,BRANCH_COVERED,LINE_MISSED,LINE_COVERED
Checkout,0,24,0,0,0,6
BulkDiscount,0,23,0,2,0,4

That is not a JaCoCo defect. Its counters are derived from bytecode, and its line rule is permissive by design: "A source line is considered executed when at least one instruction that is assigned to this line has been executed." Branches are counted "for all if and switch statements", and exception handling does not count as a branch at all.

The line rule matters more than it sounds. Run discountsLargeOrders on its own and BulkDiscount reports 4 of 4 lines, 1 of 2 branches. The ternary sits on one line, so a single call at quantity 50 marks the whole decision "executed" while one of its outcomes never ran. A line percentage cannot see that. The branch counter can, and it still cannot see that neither outcome was probed anywhere near 10.

A coverage delta answers one question: did this code execute under test. The test coverage calculator is a quick way to show a team how a percentage moves with lines rather than with behaviour. Whether to keep a generated test is a question about the requirement, asked of each test separately.

Give every generated test a clause from the ticket

Before running anything, write down which sentence of the requirement each test claims to protect, and the input that would separate a correct implementation from a plausible wrong one. For the generated suite it looks like this.

The last row is the one to reject outright. The other three are aimed at the right rule with the wrong numbers, which is cheaper to repair than it is to regenerate. If your test classes are already organised around behaviour rather than mirroring production classes, as the first part of this series argued, the "clause it claims" column is mostly a matter of reading the display names.

An expected value computed by the code it checks

Look at how discountsLargeOrders arrives at its expectation: gross.multiply(new BigDecimal("0.95")).setScale(2, RoundingMode.HALF_UP). That is the implementation, transcribed. Had the implementation been written with HALF_EVEN by mistake, a model reading it would have put HALF_EVEN in the oracle too, and the test would defend the bug. The discipline for this, a table of literals typed from a source outside the codebase, was covered in part two on parameterized tests. BigDecimal adds three traps of its own that generated oracles fall into.

The first is the double constructor. The Javadoc says plainly that "the results of this constructor can be somewhat unpredictable", and in a rounding test it can erase the exact case you meant to test:

// Wrong: 1.15 has no exact binary form. The constructor takes the double as it is,
// 1.149999999999999911182158029987476766109466552734375, so 10 units x 0.95
// lands just below the half and HALF_UP gives 10.92. The tie is gone.
BigDecimal gross = new BigDecimal(1.15).multiply(BigDecimal.valueOf(10));

// Right: the string constructor is exact. 11.50 x 0.95 = 10.9250, a true tie,
// which HALF_UP takes to 10.93 and both DOWN and HALF_EVEN take to 10.92.
BigDecimal gross = new BigDecimal("11.50");

The second is equality. BigDecimal.equals compares scale as well as value: "2.0 is not equal to 2.00 when compared by this method". When a generated test fails with expected: <95.00> but was: <95.0000>, the tempting repair is compareTo(...) == 0, and it deletes half the ticket, because "to two decimal places" is precisely a claim about scale. In this example PIT's naked-receiver mutant, which deletes the setScale call outright, is killed only because the assertion uses assertEquals.

The third works in your favour. JUnit's implicit conversion turns a CSV string into new BigDecimal("..."), so a column value of 10.93 arrives with scale 2 intact. A table of string literals keeps the scale claim honest without any extra code.

Stubbing the class that owns the rule

CheckoutTest mocks BulkDiscount, where the rule lives, and asserts that the value it stubbed comes back. The any() matcher also accepts whatever gross Checkout computes, so the one piece of arithmetic Checkout owns goes unobserved too.

Note that BulkDiscount is final, and that did not stop the model. Mockito's documentation for mocking final types states: "Since 5.0.0, this feature is enabled by default." A final modifier is no longer a signal to a generated test that a class was not meant to be replaced.

PIT names this defect precisely. Run it with only CheckoutTest selected and all four mutants in BulkDiscount.apply come back NO_COVERAGE, which the documentation defines as "the same as Survived except there were no tests that exercised the line of code where the mutation was created". The run's statistics: five mutants, one killed, four with no coverage, and a test strength of 100%.

That last figure is the trap. PIT's testStrengthThreshold is computed as "killed / (killed + survived), excluding mutants where no coverage information is available". A build gated on test strength passes this suite, because the rule the suite stubbed out has been removed from the denominator. Gate on mutationThreshold, which counts killed against all mutants, if the point of the gate is to catch tests that route around the code.

Widening the operators to the ALL group shows the cost of any() as well. Four mutants on the gross line of Checkout survive: the multiply call removed, replaced by its argument, replaced by its receiver, and swapped for divide. The stub accepts every one of those values.

The refactor test is the other reason to reject it. Fold BulkDiscount into Checkout, a change that alters no behaviour at all, and this test stops compiling, because its stub names a method that no longer exists. Stub the boundary you do not own, here the catalogue lookup, and let the rule run.

What PIT mutates in BulkDiscount, and what it never will

"Mutations are performed on the byte code generated by the compiler rather than on the source files." That one sentence from PIT's documentation explains every result in this section.

Under DEFAULTS, BulkDiscount.apply yields three mutants: the conditional boundary, the else-forcing remove-conditional, and a null return. STRONGER, which the 1.30.0 source defines as DEFAULTS plus EXPERIMENTAL_SWITCH and the two _IF remove-conditionals, adds one more. Against the generated suite only the boundary mutant survives.

That boundary mutant is the threshold mutant, just phrased differently. CONDITIONALS_BOUNDARY rewrites >= as >, and for an int quantity, quantity > 10 is the same predicate as quantity >= 11. The literal change from 10 to 11 does appear, reported as "Substituted 10 with 11", but only from INLINE_CONSTS, which sits in ALL alone. It can be mutated at all because javac copies the value of a static final int into the comparison.

The rounding rule has no operator. MATH replaces "binary arithmetic operations for either integer or floating-point arithmetic", and BigDecimal.multiply is a method call, not an arithmetic instruction. The source has an EXPERIMENTAL_BIG_DECIMAL mutator, added in 1.7.4 and missing from the documentation page, that swaps add with subtract, multiply with divide and max with min. It never touches setScale or its RoundingMode argument. The call-removal mutators delete setScale entirely, which is a different and much louder defect than choosing the wrong mode. And the MULTIPLIER field is left alone, since PIT avoids mutating static initialiser code. Across all 13 mutants ALL generates for this class, not one changes how the value is rounded.

So of the two mistakes PRICE-212 most invites, PIT hands you one for free and cannot produce the other at all.

Seeding the rounding mutant by hand

A seeded mutant is a one-line patch you apply, test against, and revert.

--- a/src/main/java/com/example/pricing/BulkDiscount.java
+++ b/src/main/java/com/example/pricing/BulkDiscount.java
@@ -13 +13 @@
-        return net.setScale(2, RoundingMode.HALF_UP);
+        return net.setScale(2, RoundingMode.DOWN);

git apply seeded/price-212-down.diff
./gradlew test --tests 'com.example.pricing.*'   # must fail
git apply -R seeded/price-212-down.diff

This is the part a model is genuinely good at. Give it the ticket text, not the implementation, and ask for the ways a competent developer could misread it, each as a one-line diff. A reasonable list for PRICE-212 is DOWN, HALF_EVEN, the threshold moved by one, and a less obvious one: rounding each unit price before multiplying by the quantity. Execution decides whether each one is killed. It cannot decide whether the per-unit reading is a mutant or an ambiguity the product owner has to settle; that call, and the check that every expected value matches the ticket's arithmetic, belong to someone who knows the domain.

Scoping PIT to the rule under review

PIT does not support JUnit Jupiter out of the box; its FAQ points to the JUnit 5 plugin, which "must be placed on the classpath of the pitest tool (not on the classpath of the project being mutated)". The Gradle plugin PIT's documentation links to does that through one property. The results above came from this configuration:

plugins {
    java
    id("info.solidsoft.pitest") version "1.19.0"
}

pitest {
    pitestVersion = "1.30.0"
    junit5PluginVersion = "1.2.3"     // adds pitest-junit5-plugin to PIT's own classpath
    targetClasses = setOf("com.example.pricing.*")
    targetTests = setOf("com.example.pricing.*")
    mutators = setOf("STRONGER")
    outputFormats = setOf("XML", "HTML")
    timestampedReports = false
}

Two details are worth knowing before the first run. The Gradle plugin reads tests from sourceSets.test by default, so an integration suite modelled as its own source set, as in part four, stays out of mutation runs unless you add it to testSourceSets. The JUnit 5 plugin has turned off Jupiter's parallel mode automatically since release 1.1.1, so a suite that only passes when run concurrently, the situation part three diagnosed, will behave differently under PIT than under ./gradlew test.

PIT writes HTML by default, XML or CSV on request, and records one of seven outcomes per mutant: killed, survived, no coverage, non viable, timed out, memory error, run error. Asked whether results are random, the FAQ answers "No", naming timeouts and static initialisers as the exceptions, which is what makes a mutation result usable as review evidence. One caution: if PIT reports zero line coverage for classes your tests obviously exercise, it is not seeing your tests, and every mutant will read as no coverage. Fix the setup before reading anything else.

The reviewed pricing tests

Two rows replace the generated BulkDiscountTest, and a real rule replaces the stub in CheckoutTest.

class BulkDiscountTest {

    private final BulkDiscount discount = new BulkDiscount();

    // PRICE-212: "10 or more units take 5% off the line; round the line
    // total HALF_UP to 2 decimal places." Expected values worked by hand.
    @ParameterizedTest
    @CsvSource(useHeadersInDisplayName = true, textBlock = """
        QUANTITY, GROSS, EXPECTED
        # one unit short of the threshold: no discount, nothing to round
        9,        10.35, 10.35
        # at the threshold: 11.50 x 0.95 = 10.925, an exact half
        10,       11.50, 10.93
        """)
    void bulkRuleStartsAtTenAndRoundsHalfUp(int quantity, BigDecimal gross, BigDecimal expected) {
        assertEquals(expected, discount.apply(gross, quantity));
    }
}

class CheckoutTest {

    @Test
    void pricesTheLineThroughTheRealDiscountRule() {
        Catalogue catalogue = sku -> new BigDecimal("1.15");   // the boundary we do not own
        Checkout checkout = new Checkout(catalogue, new BulkDiscount());

        assertEquals(new BigDecimal("10.93"), checkout.lineTotal("MUG-01", 10));
    }
}

Quantity 10 sits exactly on the threshold, so moving the threshold to 11 removes the discount there and 11.50 comes back instead of 10.93. The same row carries a true tie at the third decimal, so DOWN and HALF_EVEN both produce 10.92. Quantity 9 pins the other side, and it is what kills a threshold moved down to 9. Catalogue is an interface, so a lambda stands in for it and Mockito is not needed at all.

Text-block comments are legal here because the # is the first character on each line once Java strips the incidental indentation, a requirement the JUnit guide spells out. With useHeadersInDisplayName, the seeded DOWN mutant fails as [2] QUANTITY = "10", GROSS = "11.50", EXPECTED = "10.93" with expected: <10.93> but was: <10.92>.

The evidence to attach to the pull request links each mutant to a clause and to the row that kills it:

rule       PRICE-212  qty >= 10 takes 5% off; line total HALF_UP to 2 dp
coverage   JaCoCo lines 10/10, branches 2/2 (unchanged by this review)

mutant                               clause      before     after   killed by
CONDITIONALS_BOUNDARY  apply:12      threshold   SURVIVED   KILLED  row [2] qty 10
seeded  THRESHOLD 10 -> 11           threshold   survived   killed  row [2] qty 10
seeded  HALF_UP -> DOWN              rounding    survived   killed  row [2], CheckoutTest
seeded  HALF_UP -> HALF_EVEN         rounding    survived   killed  row [2], CheckoutTest
CheckoutTest run alone               all         NO_COVERAGE x4        real rule executes

Under STRONGER PIT kills all five mutants in the reviewed suite; under ALL, all 24.

Before the next generated JUnit test merges

Pick one class in the pull request that encodes a business rule with a number in it. Fill in the four-column table for every generated test that touches it, and reject any test whose "clause it claims" cell is empty, starting with any test that stubs the class holding the rule.

Then run PIT with targetClasses narrowed to that class and STRONGER selected. Read no-coverage results first, because each one is a rule no test reached, and a test-strength figure will hide them. Read survivors second. For every clause PIT has no operator for, ask a model for one-line misreadings of the ticket, apply each, and confirm the build fails.

The artefact to capture is the clause-to-mutant table above, committed with the tests or pasted into the review. The coverage figure can go in too, labelled as what it is: evidence that the code ran.

Questions about accepting generated JVM tests

Should the build enforce a mutation score?

On a package you have already reviewed, yes, and use mutationThreshold rather than testStrengthThreshold for the reason shown above. Mind the rounding as well. Thresholds are integer percentages by default, and PIT's Maven documentation spells out the resulting blind spot of roughly one percentage point; thresholdPrecision, added in 1.25.4, lets you compare at one or more decimal places.

Is a mocked collaborator in a generated test always wrong?

No. Mocking a boundary the class does not own, such as a catalogue service, a clock or a payment gateway, is what makes a unit test deterministic. The defect is stubbing the collaborator that computes the outcome being asserted. The quick check: if deleting the production class under test would leave the test green, the mock is in the wrong place. PIT reports exactly that case as no coverage.

Can the model write the expected values if it is given the ticket instead of the code?

It can propose them, which beats transcribing the implementation. A person still redoes the arithmetic, because a hand-worked oracle is only worth having if it comes from outside the chain that produced the code. Checking 11.50 × 0.95 = 10.925 takes seconds, and that check is the review.

Does this work for Kotlin code on the JVM?

Not with stock PIT. Its FAQ warns that bytecode from languages it does not explicitly support yields "junk" mutants, and lists Kotlin as supported only through the Arcmutate Kotlin plugin. The review table and hand-seeded mutants work in any language.

Primary references

• PIT — Mutation operators. That mutations "are performed on the byte code generated by the compiler rather than on the source files"; the CONDITIONALS_BOUNDARY replacement table (>= to >); that MATH covers "binary arithmetic operations for either integer or floating-point arithmetic"; INLINE_CONSTS, NON_VOID_METHOD_CALLS, the experimental naked-receiver and argument-propagation mutators; and the group table that still places Negate Conditionals in DEFAULTS.
• PIT 1.30.0 source — StandardMutatorGroups.java. The actual membership of DEFAULTS and STRONGER at the current release, which differs from the documentation table.
• PIT 1.30.0 source — BigDecimalMutator.java. That EXPERIMENTAL_BIG_DECIMAL swaps add/subtract, multiply/divide, max/min and negate/plus, and has no replacement for setScale or rounding modes.
• PIT README at 1.30.0 — release notes. 1.7.2 "Change default mutators - replace negate conditional with remove conditional", 1.7.4 adding the BigDecimal mutator, and 1.25.4 adding configurable decimal precision. The 1.30.0 release page explains the jump from 1.25.9.
• PIT — Basic concepts. The seven mutant outcomes, and that "no coverage is the same as Survived except there were no tests that exercised the line of code where the mutation was created".
• PIT — Maven quick start. targetClasses and targetTests as globs, outputFormats (HTML, XML, CSV; HTML by default), mutationThreshold, testStrengthThreshold defined as "killed / (killed + survived), excluding mutants where no coverage information is available", and the integer threshold blind spot with thresholdPrecision.
• PIT — FAQ. That JUnit 5 needs a plugin, that PIT avoids mutating static initialisers, the DEFAULTS/STRONGER/ALL groups with ALL "strongly discouraged", that results are deterministic apart from timeouts and static initialisers, the advice to limit analysis to changed code, and Kotlin support only via Arcmutate.
• pitest-junit5-plugin — README. Release 1.2.3, that the plugin must sit on PIT's classpath rather than the project's, and that 1.1.1 automatically disables parallel mode.
• gradle-pitest-plugin — README. The info.solidsoft.pitest plugin at 1.19.0, junit5PluginVersion, pitestVersion, the Kotlin DSL property assignment form, and testSourceSets defaulting to sourceSets.test.
• JaCoCo — Coverage Counters. That counters derive from bytecode, the line rule "a source line is considered executed when at least one instruction that is assigned to this line has been executed", branch counting for if and switch, and that exception handling is not a branch. JaCoCo 0.8.15 is the current release.
• Java SE 27 — BigDecimal. That equals considers scale, so "2.0 is not equal to 2.00"; that the double constructor's results "can be somewhat unpredictable"; and setScale(int, RoundingMode) applying the rounding mode when the scale is reduced.
• Java SE 27 — RoundingMode. HALF_UP rounding up when "both neighbors are equidistant", HALF_EVEN behaving as HALF_UP only when the digit left of the discarded fraction is odd, and DOWN never incrementing the retained digit.
• Mockito 5.24.0 — Mockito class Javadoc. Section 39: mocking final types and methods, "Since 5.0.0, this feature is enabled by default"; and the argument-matcher behaviour of any() and anyInt().
• Parameterized classes and tests, in the JUnit 6.1.3 User Guide. @CsvSource with textBlock and useHeadersInDisplayName, the display-name format, comment lines requiring # as the first character, and implicit conversion of a string to new BigDecimal(...).