# Review AI-Generated pytest Tests with Mutation Checks

> A generated test file can execute every branch of a pricing rule and still pass when the threshold moves by one, because coverage records that a line ran and nothing records that an assertion depended on it. This guide covers scoping a mutmut run to a single function, telling a surviving mutant apart from a mutant that any calling test kills, the two review patterns — a mock placed one layer too low and an assertion that restates its own fixture — that make every mutant survive, and the pragmas and settings that quietly remove code from the run.

- Author: [Shashank Rawlani](https://shashank.rawlani.com)

- Published: 2026-09-25T13:30:00.000Z

- Updated: 2026-10-14T11:09:52.936Z

- Category: AI in Testing

- Canonical source: https://automationtester.in/blog/ai-in-testing/review-ai-generated-pytest-tests-mutation-checks

- Tags: python, pytest, mutation-testing, ai-assisted-testing, code-review, test-design

**Quick answer:** Coverage proves a line ran. A surviving mutant proves no assertion depended on it. Change one character of the behaviour under review: `<` to `<=`, or a threshold literal by one, and rerun only the tests that touch it. Any generated test that still passes is asserting something other than the rule it claims to protect, and the review note writes itself: the mutant, the test that should have caught it, and the boundary value it never used.

Here is a pull request that a reviewer approves in forty seconds. A pricing rule, and two tests for it:

```
# pricing.py
from decimal import Decimal

BULK_THRESHOLD = 10

def discount_for(quantity: int, unit_price: Decimal) -> Decimal:
    if quantity < BULK_THRESHOLD:
        return Decimal("0")
    return (unit_price * quantity * Decimal("0.15")).quantize(Decimal("0.01"))
```

```
# test_pricing.py  — generated
def test_discount_applied_for_bulk_order():
    assert discount_for(50, Decimal("2.00")) == Decimal("15.00")


def test_no_discount_for_small_order():
    assert discount_for(5, Decimal("2.00")) == Decimal("0")
```

Both branches execute. Statement and branch coverage of `discount_for` are both 100%. And the pair cannot detect the single most likely defect in the function, which is that the threshold is off by one.

Move `BULK_THRESHOLD` to 11. Both tests still pass: 5 is below either threshold and 50 is above either. Change `<` to `<=`. Both tests still pass, for the same reason. The rule these tests exist to defend, "bulk pricing starts at ten units," is the one thing they never look at.

**A clean mutation report is only as strong as the mutants that were generated.** mutmut 3 does not mutate code outside functions at all, so a threshold defined at module scope produces no mutant. With `mutate_only_covered_lines` enabled it also skips anything your coverage configuration excludes, and `# pragma: no mutate` plus `do_not_mutate_patterns` can remove more. Read those settings as part of the diff before you read the score. Checked against mutmut 3.7.0 and coverage.py 7.16.0 on 2026-09-07.

## A surviving mutant is a missing assertion

Mutation testing makes that argument mechanical. A mutation tool edits your source, runs the tests, and records whether anything failed. A mutant that gets caught is *killed*; one that does not is a *survivor*. Every survivor is a change to production behaviour that your suite would have merged.

mutmut 3.7 keeps its mutation catalogue deliberately small and deliberately subtle. The three you will meet on almost every review are stated outright in its documentation:

- Integer literals are incremented: "so 0 becomes 1, 5 becomes 6, etc."

- `<` is changed to `<=`.

- `break` is changed to `continue` and vice versa.

Those are exactly the edits a tired author makes by accident, which is the point: "in general the idea is that the mutations should be as subtle as possible." Run it against the function above and you get two survivors and one kill:

```
$ mutmut run "pricing.discount_for*"
$ mutmut browse
```

```
pricing.discount_for
  x_discount_for__mutmut_1   BULK_THRESHOLD -> 11        survived
  x_discount_for__mutmut_2   < -> <=                     survived
  x_discount_for__mutmut_3   "0.15" -> "XX0.15XX"        killed
```

The kill is the one worth ignoring. mutmut mutates string literals by wrapping them. The documentation's own example is `logging.info("Foo")` becoming `logging.info("XXFooXX")`, so `Decimal("XX0.15XX")` raises `InvalidOperation` before any assertion runs. Any test that calls the function at all kills that mutant. It tells you the function was invoked, which is what coverage already told you.

Sort survivors and trivial kills apart before you write a single review comment. A suite can have a high mutation score built almost entirely from mutants that crash on contact, and the two survivors above are worth more than fifty of those.

## One function at a time, not the repository

Mutation runs are usually abandoned because someone started with `mutmut run` across a whole service and watched it for an hour. Reviewing a pull request does not need that. mutmut takes Unix filename patterns, and the unit can be a single function:

```
mutmut run "pricing*"                    # one module
mutmut run "pricing.discount_for*"       # one function
```

Scope it to the functions the pull request touched. That turns a mutation run from an overnight job into something that finishes while you read the diff, and it keeps the result attributable: a survivor in a function nobody changed is not this author's problem.

Two configuration choices decide whether the run is usable at all. Both live in `pyproject.toml`:

```
[tool.mutmut]
source_paths = [ "src/" ]
pytest_add_cli_args_test_selection = [ "tests/unit" ]
# Only mutate lines coverage.py saw execute. Default is false, which means
# mutmut falls back to "any function that gets called".
mutate_only_covered_lines = true
# Stop counting a test as relevant to a function once the call is this deep.
max_stack_depth = 8
```

`max_stack_depth` is the one people miss, and mutmut is explicit about the trade it makes: a lower value "will increase mutation speed and lead to more localized tests, but will also lead to more surviving mutants that would otherwise have been caught." Without it, a utility called incidentally from three hundred integration tests drags all three hundred into every mutant's verdict, and a survivor stops telling you which test was supposed to catch it.

The other number to know before you start is the timeout. mutmut kills a slow mutant after `(duration_of_original_tests + timeout_constant) * timeout_multiplier` seconds, defaulting to a constant of 1.0 and a multiplier of 15.0. That formula is multiplicative in your suite's own runtime, so a selection containing one test that sleeps for two real seconds does not cost two seconds per mutant. It moves the ceiling for every mutant in the run. Point `pytest_add_cli_args_test_selection` at the fast unit tests and leave the sleeping ones out.

## The mock that deletes the code under test

When every mutant in a function survives, not one, all of them, the function almost certainly never ran. The usual cause in generated tests is a mock placed one layer too low:

```
# Wrong. checkout.discount_for is the behaviour this test claims to cover,
# and this line replaces it. Every mutant inside discount_for survives.
def test_checkout_applies_bulk_discount(monkeypatch):
    monkeypatch.setattr("checkout.discount_for", lambda q, p: Decimal("15.00"))

    assert checkout_total(50, Decimal("2.00")) == Decimal("85.00")
```

```
# Right. Mock the boundary the test does not own — the tax service — and let
# the pricing rule execute. Mutants inside discount_for now have somewhere
# to show up.
def test_checkout_applies_bulk_discount():
    tax = create_autospec(TaxService, instance=True)
    tax.rate_for.return_value = Decimal("0")

    assert checkout_total(50, Decimal("2.00"), tax) == Decimal("85.00")
```

Use `create_autospec` or `autospec=True` rather than a bare `Mock`, because the standard library is blunt about what a bare one costs: "if you refactor some of your code, rename members and so on, any tests for code that is still using the old api but uses mocks instead of the real objects will still pass. This means your tests can all pass even though your code is broken." Autospec fixes that by limiting the mock to the real object's API and giving mocked methods "the same call signature as the original so they raise a `TypeError` if they are called incorrectly."

The related failure is a patch that silently does nothing. `unittest.mock`'s rule is that "you patch where an object is looked up, which is not necessarily the same place as where it is defined". If `checkout.py` says `from pricing import discount_for`, then patching `pricing.discount_for` has no effect, because `checkout` already holds a reference to the real function. The test passes, the mock is never consulted, and nothing in the output says so. A mutation run does say so: the mutants get killed by a test the author believed was stubbed out.

## Assertions that only restate the fixture

The second pattern that produces uniform survival is an assertion whose expected value came from the setup two lines above it:

```
def test_order_totals(order_factory):
    order = order_factory(subtotal=Decimal("100.00"), currency="USD")

    assert order.subtotal == Decimal("100.00")   # the factory said so
    assert order.currency == "USD"               # the factory said so
    assert order.total > 0                       # true for every non-empty order
```

Nothing here depends on the pricing rule, so no mutation of it can fail this test. The discriminating question at review time is not "is this assertion true" but "what value would make it false". If the answer is "only a broken factory", the test belongs to the factory, not to the order.

Rewrite it with a value the test computed independently of the code path it is checking:

```
@pytest.mark.parametrize(
    ("quantity", "expected"),
    [
        pytest.param(9, Decimal("0"), id="one-below-threshold"),
        pytest.param(10, Decimal("3.00"), id="exactly-at-threshold"),
        pytest.param(11, Decimal("3.30"), id="one-above-threshold"),
    ],
)
def test_bulk_discount_starts_at_ten_units(quantity, expected):
    assert discount_for(quantity, Decimal("2.00")) == expected
```

Those three rows kill both survivors from earlier. The 9/10 pair kills the threshold-literal mutant, and the same pair kills the `<`-to-`<=` mutant, because moving the comparison moves the boundary between them. The 11 row is not redundant padding: it is what distinguishes "the boundary is right" from "the boundary is right and the arithmetic still scales".

Note that the expected figures were derived from the written rule: fifteen per cent of quantity times unit price, rounded to the cent, and not from running the function and pasting what it returned. An oracle copied from the implementation cannot disagree with the implementation, which is the failure mutation testing is least able to help with.

## What the tool quietly declines to mutate

A clean mutation report is only as strong as the set of mutants that were generated, and mutmut skips more than most people expect.

Module-level code is not mutated at all in mutmut 3. The documentation points you at mutmut 2, "which has a different execution model", if you need it. Constants defined at import time, dataclass field defaults and configuration dictionaries therefore produce no mutants, and a threshold that lives at module scope rather than inside the function is invisible to the run.

With `mutate_only_covered_lines = true`, mutmut "also honours the lines coverage.py is told to leave out of its measurement, so code marked with `# pragma: no cover`, or matched by your `exclude_lines`/`exclude_also` settings, is not mutated either." An exclusion added to make a coverage gate green silently removes that code from mutation as well.

And mutation can be turned off in the source. `# pragma: no mutate` suppresses a line, `# pragma: no mutate block` suppresses a whole indentation block including an entire function when placed inline on the `def`, and `do_not_mutate_patterns` suppresses by regex across the codebase. All three are legitimate: the documented use is logging calls and `break`-versus-`continue` optimisations. All three are also a way to make a report look clean. Read them as part of the diff.

```
# Worth a look before trusting any mutation number in a pull request.
git diff --unified=0 origin/main -- '*.py' | grep -E 'pragma: no (mutate|cover)'
git diff origin/main -- pyproject.toml setup.cfg | grep -E 'do_not_mutate|only_mutate|max_stack_depth'
```

## Where branch coverage still earns its place

None of this retires coverage. Branch coverage answers a question mutation testing does not: which paths were never taken at all. Coverage.py tracks, for each line that can jump to more than one next line, which destinations were actually visited, and reports the missing transition by number: in its own `if`/`else` example, a flag that is always true reports the missing branch as `2->5`.

That is the right first pass on a generated test file, and it is cheap. Run it with `--branch`, look for partial branches in the code the pull request touched, and treat each one as a case the author did not think of. Then mutate the branches that *were* taken, because that is the set coverage has already declared satisfied and cannot say anything more about. The two tools disagree about different things, which is why running one is not a reason to skip the other.

## Apply this now

Take the next pull request containing generated tests and pick the one function in it that encodes a business rule with a number in it: a threshold, a limit, a retention window, a rate band. Run `mutmut run "<module>.<function>*"` against it with the fast unit selection only.

Then split the report in two. Trivial kills, where the mutated source raises before an assertion runs, go in one pile and are not evidence. Survivors go in the other, and each one becomes a review comment of exactly three lines: the mutant mutmut printed, the test whose name claims to cover that rule, and the input value that would have separated them.

What you should have at the end is a diff that adds boundary rows rather than test functions, and a short list of rules whose thresholds are now pinned by a test that fails when they move. If the mutation run produced no survivors and no interesting kills either, check `source_paths` and the pragmas before believing it.

## Questions that come up in review

### Is there a mutation score worth enforcing in CI?

Not as a global gate, at least not first. A repository-wide percentage is dominated by trivial kills and by code nobody disputes, so it moves for reasons unrelated to test quality and teams learn to ignore it. mutmut does support publishing one. `mutmut export-cicd-stats` feeds `mutmut badge`, which emits Shields endpoint JSON, and that is genuinely useful as a trend on a module you are actively hardening. Enforce it on a package you have already cleaned, not on the whole tree on day one.

### How is this different from generating more inputs?

They find opposite things. Property-based generation searches the input space for a case your assertions reject; mutation testing holds the inputs fixed and searches the *source* for a change your assertions ignore. A property test with a weak property will happily explore ten thousand inputs and kill nothing, and a mutation run will tell you so. Where they meet is useful: a surviving mutant often names the input dimension your generator was never given a strategy for.

### The run takes too long on our suite. What is the first thing to change?

The test selection, not the mutation settings. mutmut runs your chosen tests once per mutant, so anything slow is multiplied by the mutant count, and the timeout ceiling scales with your suite's own runtime as well. Point `pytest_add_cli_args_test_selection` at unit tests only, add `max_stack_depth`, and scope the run with a pattern. Reach for parallelism after those three, because they change the size of the problem rather than the speed of the machine.

### Should a generated test ever be deleted outright?

Yes, and the survivors tell you which. A test that kills no mutant that another test does not already kill, and whose assertions all restate its own setup, is costing you runtime and a maintenance obligation in exchange for nothing. Deleting it is not a reduction in coverage of behaviour, because it was covering none. The ones to keep and repair are those aimed at a real rule with the wrong assertion attached.

### Does any of this work without installing a mutation tool?

The technique does, at the cost of doing it by hand. Change the threshold literal by one in your working copy, run the tests, and change it back. If nothing fails, the review comment is the same one you would have written from a report. That is worth doing on the spot for a single suspicious rule, and it is how you convince a team that the tool is worth adding: a live demonstration that the suite they trust does not notice the change is more persuasive than a percentage.

## Primary references

- [mutmut — documentation](https://mutmut.readthedocs.io/en/latest/). The mutation catalogue stated in "Example mutations": integer literals incremented, `<` changed to `<=`, `break` and `continue` swapped, with the design goal that "the mutations should be as subtle as possible". Also the filename-pattern scoping (`mutmut run "my_module.my_function*"`), the `source_paths` and `pytest_add_cli_args_test_selection` configuration, `mutate_only_covered_lines` and its interaction with `# pragma: no cover`, `max_stack_depth` and its stated trade-off, the `no mutate` pragmas and `do_not_mutate_patterns`, the timeout formula `(duration_of_original_tests + timeout_constant) * timeout_multiplier` with defaults 1.0 and 15.0, the string-literal mutation example `logging.info("XXFooXX")`, that mutmut 3 does not mutate code outside functions, and `export-cicd-stats` with `mutmut badge`.

- [mutmut on PyPI](https://pypi.org/project/mutmut/). The current release, 3.7.0, and the fork requirement that means Windows use goes through WSL, checked 2026-09-07.

- [Python — unittest.mock: Autospeccing](https://docs.python.org/3/library/unittest.mock.html#autospeccing). That a plain mock survives a rename in the code under test, so "your tests can all pass even though your code is broken", and that autospecced functions "have the same call signature as the original so they raise a `TypeError` if they are called incorrectly". Also that a `spec` turns an attribute typo into an `AttributeError` rather than a silently passing assertion.

- [Python — unittest.mock: Where to patch](https://docs.python.org/3/library/unittest.mock.html#where-to-patch). That "you patch where an object is looked up, which is not necessarily the same place as where it is defined", and the worked `from a import SomeClass` case where patching the defining module has no effect at all.

- [Coverage.py — Branch coverage measurement](https://coverage.readthedocs.io/en/latest/branch.html). What a partial branch is, that the coverage percentage is "actual executions divided by the execution opportunities" with each branch destination counting as one, and the reporting form for a missing transition such as `2->5` in the `if`/`else` example. Version 7.16.0.

- [pytest — How to parametrize fixtures and test functions](https://docs.pytest.org/en/stable/how-to/parametrize.html). The `@pytest.mark.parametrize` signature and `pytest.param(..., id=...)`, which is what lets each boundary row report under a name a reviewer can read in the failure output.

## Continue reading on AutomationTester.in

- [Reviewing AI-Generated Playwright Tests: Eight Tells in the Diff](https://automationtester.in/blog/ai-in-testing/review-ai-generated-playwright-tests-before-merge)
- [Validate AI Agent Tool Calls in TypeScript: Schema, Authorization, Approval](https://automationtester.in/blog/ai-in-testing/validate-ai-agent-tool-calls-typescript)
- [Review AI-Generated JUnit Tests Before They Raise Coverage](https://automationtester.in/blog/ai-in-testing/review-ai-generated-junit-tests-before-coverage)
- [Review an AI-Generated OpenAPI Description Before You Publish It](https://automationtester.in/blog/ai-in-testing/review-ai-generated-openapi-specifications)

Source: [Review AI-Generated pytest Tests with Mutation Checks](https://automationtester.in/blog/ai-in-testing/review-ai-generated-pytest-tests-mutation-checks) by Shashank Rawlani.