Use pytest Parametrization Without Hiding Test Intent

A build fails with one red row out of 340. The row is called test_validates_payload[a0-b0-expected0].

Nothing in that string says which validation rule broke. Nothing says how to run it again. The ID is a position, so reproducing it means opening the file and counting entries in a list. And by the time someone has counted, another case has been appended to the table and the numbers have shifted.

Parametrization is the cheapest way to add coverage in pytest and the cheapest way to lose the ability to act on it. The two properties are not in tension; the second is a consequence of leaving the ID to be generated.

What pytest puts in the brackets

The rule is short and worth knowing exactly, because it explains every unreadable report you have seen. From the examples page: "Numbers, strings, booleans and None will have their usual string representation used in the test ID. For other objects, pytest will make a string based on the argument name."

So a table of primitives reads fine and a table of objects does not:

testdata = [
    (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
    (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]

@pytest.mark.parametrize("a,b,expected", testdata)
def test_timedistance_v0(a, b, expected):
    assert a - b == expected

$ pytest test_time.py --collect-only
    <Function test_timedistance_v0[a0-b0-expected0]>
    <Function test_timedistance_v0[a1-b1-expected1]>

Three things follow from this, and the third is the one people miss. The ID identifies the failing case in the report. The ID is what -k matches against, so it is also your selection handle. And --collect-only prints the IDs without running anything, which makes it the fastest way to see what your table actually produced. Run it once against your worst parametrized test before changing any code.

Attach the name to the data, not to a parallel list

pytest offers four ways to control the ID, and they are not equivalent under maintenance.

A list of strings is the obvious one and the one that rots. It is positional, so it is correct only while nobody inserts a case in the middle:

# Works, but the names and the data are now two lists that must stay
# aligned by hand. Add a case to testdata and every name shifts by one.
@pytest.mark.parametrize("a,b,expected", testdata, ids=["forward", "backward"])
def test_timedistance_v1(a, b, expected):
    assert a - b == expected

A callable derives the ID from the value. Its useful property is that it is partial: pytest documents that "returning None will use an auto-generated id," and that a callable's return value "is used as part of the auto-generated id for the whole set (where parts are joined with dashes)." So you can label the argument that matters and leave the rest alone:

def idfn(val):
    if isinstance(val, datetime):
        return val.strftime("%Y%m%d")
    # No return for timedelta: those keep pytest's default representation.

@pytest.mark.parametrize("a,b,expected", testdata, ids=idfn)
def test_timedistance_v2(a, b, expected):
    assert a - b == expected

# Produces: test_timedistance_v2[20011212-20011211-expected0]
#                                                  ^^^^^^^^^ still auto

pytest.param is the one to reach for by default. It puts the name in the same expression as the data, so the two cannot drift apart:

@pytest.mark.parametrize(
    "payload,expected_error",
    [
        pytest.param({}, "email_required", id="email-missing"),
        pytest.param({"email": "a" * 321}, "email_too_long", id="email-over-320-chars"),
        pytest.param({"email": "not-an-email"}, "email_malformed", id="email-no-at-sign"),
        pytest.param({"email": " [email protected] "}, None, id="email-trimmed-and-lowercased"),
    ],
)
def test_email_validation(payload, expected_error):
    assert validate(payload).error == expected_error

Read the four IDs on their own: email-missing, email-over-320-chars, email-no-at-sign, email-trimmed-and-lowercased. Each names the rule it protects rather than the data it uses. That is the property to aim for. A stranger reading the CI summary should learn what broke without opening the test file.

The same call takes marks, which is how a known-broken case stays visible instead of being commented out:

pytest.param(
    {"email": "[email protected]"}, None,
    id="email-plus-addressing",
    marks=pytest.mark.xfail(reason="AUT-418: plus addressing rejected", strict=True),
)

Since pytest 8.4 there is also pytest.HIDDEN_PARAM, which omits a parameter set from the test name entirely and may be used at most once per test because names must stay unique. It is narrow, useful when one argument is a fixture-shaped implementation detail that adds nothing to the report.

Collisions are renumbered behind your back

This is the default that turns IDs into positions. When two parameter sets generate the same ID, pytest does not complain. The reference is explicit: if strict_parametrization_ids is not set, "pytest automatically handles this by adding 0, 1, … to duplicate IDs, making them unique."

Two consequences, both of which show up weeks later. A saved command like pytest "tests/test_api.py::test_limits[100]" can silently start selecting a different case, because the case it named was 100 and there are now 1000 and 1001. And any tool keyed on the node ID (flake trackers, quarantine lists, historical pass rates) attributes the new case's failures to the old case's history.

Turn the check on. It is a collection-time error, so it costs one run to find every collision you already have:

[pytest]
strict_parametrization_ids = true

With it enabled, @pytest.mark.parametrize("letter", ["a", "a"]) is an error rather than two tests called a0 and a1. If the duplicate is intentional, the documented fix is to name the cases yourself, which forces you to say why there are two, and that sentence is usually the missing piece.

The table is evaluated during collection

A parameter list is an expression in a decorator. It runs when the module is imported, before any test executes, for every case whether or not that case will run. Anything with an effect that sits in the table happens at collection.

# Wrong. create_customer() is called four times during collection —
# including when you run pytest -k email-missing, which executes one
# test. Collection also happens under --collect-only, so listing your
# tests provisions four customers.
@pytest.mark.parametrize(
    "customer,expected",
    [
        pytest.param(create_customer(plan="free"), "upgrade_required", id="free-plan"),
        pytest.param(create_customer(plan="pro"), None, id="pro-plan"),
    ],
)
def test_plan_gate(customer, expected): ...

The documented route out is indirect, which passes the parameter to a fixture instead of to the test. pytest names this use case directly: it lets you "do more expensive setup at test run time in the fixture, rather than having to run those setup steps at collection time."

# Right. The table holds a plan name — inert data. The fixture turns it
# into a customer when the test actually runs, and can tear it down.
@pytest.fixture
def customer(request, api_client):
    created = api_client.create_customer(plan=request.param)
    yield created
    api_client.delete_customer_if_present(created.id)

@pytest.mark.parametrize(
    "customer,expected",
    [
        pytest.param("free", "upgrade_required", id="free-plan"),
        pytest.param("pro", None, id="pro-plan"),
    ],
    indirect=["customer"],
)
def test_plan_gate(customer, expected): ...

Note indirect=["customer"] rather than indirect=True. Passing a list applies indirection to the named arguments only, so expected reaches the test as a plain value while customer is routed through the fixture. indirect=True would try to resolve both through fixtures and fail on the one that has none.

There is a second collection-time trap in the same area, and pytest states it in a note on the parametrize page: "Parameter values are passed as-is to tests (no copy whatsoever)." It spells out the consequence. If you pass a list or dict and the test mutates it, "the mutations will be reflected in subsequent test case calls."

BASE = {"email": "[email protected]", "country": "IN"}

# Wrong. Both cases receive the same dict object. The first test pops
# 'country', so the second one sees a payload it was never given —
# and it passes or fails depending on collection order.
@pytest.mark.parametrize(
    "payload,expected",
    [pytest.param(BASE, None, id="complete"), pytest.param(BASE, None, id="repeat")],
)
def test_accepts_payload(payload, expected):
    payload.pop("country", None)
    assert submit(payload).error == expected

The fix is not to copy defensively inside every test. It is to keep mutable state out of the table: parametrize over the scalar that varies, and build the payload inside the test or in a fixture, where each case gets its own object.

Pick equivalence classes before you pick values

Stacked parametrize decorators multiply. pytest describes the behaviour precisely. Stacking produces every combination, "exhausting parameters in the order of the decorators":

@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass
# x=0/y=2, x=1/y=2, x=0/y=3, x=1/y=3

Two arguments is fine. The failure is what happens when a third and fourth get added by different people: four currencies × three plans × three countries × two payment methods is 72 cases, and they are testing four rules. Runtime is the smaller cost. The real cost is that a rule change now breaks eleven cases, and nobody can tell whether that is one bug or eleven.

Work the other direction. Name the classes the code actually branches on, then choose one value per class plus the boundaries:

# The rule under test: VAT applies above a threshold, in EU countries,
# for non-exempt plans. Three classes, and the threshold has two edges.
@pytest.mark.parametrize(
    "country,plan,cents,expected_vat",
    [
        pytest.param("DE", "pro", 10_000, 1_900, id="eu-above-threshold"),
        pytest.param("DE", "pro",  1_999,     0, id="eu-one-cent-below-threshold"),
        pytest.param("DE", "pro",  2_000,   380, id="eu-exactly-at-threshold"),
        pytest.param("IN", "pro", 10_000,     0, id="non-eu-never-charged"),
        pytest.param("DE", "edu", 10_000,     0, id="eu-exempt-plan"),
    ],
)
def test_vat(country, plan, cents, expected_vat):
    assert vat_for(country, plan, cents) == expected_vat

Five cases, five reasons, and the two threshold cases sit next to each other so the boundary is visible as a boundary. A product would have generated this and forty-five others, and the forty-five would have been indistinguishable from each other in the report.

The cost of doing it this way is honest and worth stating: named classes are a claim about how the code branches, and when the rule changes, the IDs become wrong before the assertions do. An ID that says eu-exactly-at-threshold after the threshold moved is worse than no name. Reviewing IDs when the rule changes is part of the maintenance you are taking on.

An empty table produces a passing run

If the parameter list is built by a function (reading a fixtures directory, querying a schema, filtering a catalogue), it can come back empty. pytest's default response is to skip, not to fail. The empty_parameter_set_mark option governs it, with three documented values: skip (the default) "skips tests with an empty parameterset," xfail marks them xfail(run=False), and fail_at_collect "raises an exception if parametrize collects an empty parameter set."

The default is the dangerous one for generated tables, because a skip is green. A glob that stops matching after a directory rename removes an entire test file's worth of coverage and reports it as a skip count nobody reads.

[pytest]
# A generated table that came back empty is a bug in the generator,
# not a reason to skip. pytest notes the default is planned to change
# to xfail in a future release for the same reason.
empty_parameter_set_mark = "fail_at_collect"

Apply this now

Run pytest --collect-only -q and pipe it through a filter for auto-generated IDs: anything matching an argument name followed by a digit, such as [a0-b0] or [payload3]. Every hit is a case whose report line cannot be read or rerun. Those are your rewrite list, ordered by how often the file appears in your failure history.

Then add both settings to your config in one commit, because each one converts a silent condition into a collection error and you want to find them together:

[pytest]
strict_parametrization_ids = true
empty_parameter_set_mark = "fail_at_collect"

You will know it landed when the CI summary is actionable without opening an editor: a failure named for the rule it broke, and a copy-pasteable node ID that still selects the same case a month later.

Questions about parametrized case design

How do I rerun exactly one parametrized case?

Quote the full node ID, brackets included: pytest "tests/test_billing.py::test_vat[eu-exactly-at-threshold]". The quotes matter because brackets are shell globs. -k also matches against the ID and is better when you want a group (-k "eu-" selects every EU case), but it is a substring match, so it will happily pick up cases you did not intend once the suite grows.

Should parametrize data live in a YAML or CSV file?

Only when a non-engineer genuinely maintains it, and then keep the ID in the file as a named column rather than deriving it from the row index. Moving the table out of Python costs you the two things this article is about: the IDs become whatever the loader generates, and marks is no longer available, so a known-broken row has to be deleted or filtered instead of marked xfail. If the data is maintained by the same people who maintain the test, the file buys nothing.

Why do non-ASCII IDs appear escaped?

Because pytest escapes them by default, and it says why the alternative is discouraged in the name of the option that disables it: disable_test_id_escaping_and_forfeit_all_rights_to_community_support. The docs warn it "might cause unwanted side effects and even bugs depending on the OS used and plugins currently installed." If you are testing internationalisation, the better move is an explicit ASCII ID describing the case (id="devanagari-name") and keep the actual string in the data where it belongs.

Can a parametrized fixture and a parametrized test be combined?

Yes, and the result is their product, which is usually not what was wanted. Before adding a second axis, check whether the fixture's parameters represent the same equivalence class as the test's. A fixture parametrized over three database backends crossed with a test parametrized over five validation rules gives fifteen cases to protect five rules against one variable. Parametrizing the fixture with scope is worth knowing about here: the reference notes it "will also override any fixture-function defined scope," so the grouping of tests by parameter instance is something you control rather than inherit.

Is it worth parametrizing a test with two cases?

Often not. Two cases that share one assertion are a parametrize; two cases that need different assertions are two tests with names, and two named test functions read better in a report than one function with two IDs. The signal that you have gone too far is a test body containing if expected_error is None. At that point the parameter is selecting behaviour rather than data, and splitting the test is the smaller change.

Primary references

• pytest — Parametrizing tests: Different options for test IDs. That numbers, strings, booleans and None use their usual string representation in the ID while other objects produce a string based on the argument name; that IDs are what -k selects on; and that --collect-only shows the generated IDs.
• pytest — Configuration: strict_parametrization_ids. That the option defaults to false, and that when it is unset pytest "automatically handles" duplicate IDs "by adding 0, 1, … to duplicate IDs, making them unique" — plus the documented fix of assigning explicit IDs.
• pytest — Configuration: empty_parameter_set_mark. The default of skip, the three accepted values including fail_at_collect, and the note that the default is planned to change to xfail "as this is considered less error prone".
• pytest — How to parametrize: @pytest.mark.parametrize. That "parameter values are passed as-is to tests (no copy whatsoever)" and mutations are "reflected in subsequent test case calls"; that stacking decorators yields every combination "exhausting parameters in the order of the decorators"; and the escaping option and its warning.
• pytest — Indirect parametrization. That indirect=True passes the value to a fixture as request.param, and that this exists to "do more expensive setup at test run time in the fixture, rather than having to run those setup steps at collection time".
• pytest — Apply indirect on particular arguments. That passing a list or tuple of argument names to indirect routes only those arguments through fixtures while the rest reach the test directly.
• pytest — API reference: pytest.param. The id and marks arguments, that usefixtures cannot be added through marks, and that pytest.HIDDEN_PARAM (added in 8.4) hides a parameter set from the test name and may be used at most once.
• pytest — API reference: Metafunc.parametrize. That an ids callable contributes "part of the auto-generated id for the whole set (where parts are joined with dashes)" and that returning None falls back to the auto-generated id; and that scope "will also override any fixture-function defined scope".