Property-Based Testing in Python: Start with Invariants

Here is a property test that has run in CI for six months, a hundred inputs per run, and has never once failed:

@given(st.text())
def test_encode_returns_a_string(value):
    assert isinstance(url_encode(value), str)

It is not a property test. It is a type annotation with a random number generator attached. There is no input in the domain that could falsify it, so the sixty thousand inputs it has consumed have established exactly one thing: that url_encode returns a string.

Meanwhile the real defect, a lone surrogate character that survives encoding and comes back as something else, was never generated, and could not have been, because the default alphabet excludes surrogates on purpose. The test was not weak because Hypothesis is weak. It was weak because nobody wrote down what had to be true and which inputs counted.

A property is a claim something could violate

The usable test for whether you have a property is short: name an input that would make it false. If you cannot, you have an assertion about types or a restatement of the implementation.

Three families reliably produce claims that can fail, and they need different amounts of work from you.

Round-trip is the cheapest and the strongest where it applies, because the oracle is the input itself:

@given(st.text())
def test_url_encoding_round_trips(value):
    assert url_decode(url_encode(value)) == value

Metamorphic properties relate two runs of the same function, which is what you reach for when there is no inverse. Each of these can fail independently, and each names a different mistake:

@given(st.lists(st.integers()))
def test_sorting_is_idempotent(xs):
    assert sorted(sorted(xs)) == sorted(xs)

@given(st.lists(st.integers()))
def test_sorting_preserves_multiset(xs):
    assert Counter(sorted(xs)) == Counter(xs)          # nothing invented or lost

@given(st.lists(st.integers()), st.integers())
def test_insertion_keeps_order(xs, y):
    result = insert_sorted(sorted(xs), y)
    assert all(a <= b for a, b in zip(result, result[1:]))

Differential properties compare your implementation against something you already trust: the previous version during a rewrite, a slow reference implementation, or the library you are replacing. This is the highest-value family for migrations and the only one that needs no invariant of its own, because the oracle supplies it.

What none of these are is a parametrized test with extra steps. A property that reads assert encode("a b") == "a%20b" under @given is an example that ignores its input, and Hypothesis will burn a hundred generations proving it.

The default alphabet is a decision

This is the fact that makes the opening test hollow, and it is stated plainly in the strategies reference. The signature of text() is text(alphabet=characters(codec='utf-8'), *, min_size=0, max_size=None), and the documentation explains what that default excludes: "the default alphabet strategy can generate the full unicode range but excludes surrogate characters because they are invalid in the UTF-8 encoding."

It then tells you what to do about it: "you can use characters() without arguments to find surrogate-related bugs such as bpo-34454."

# Cannot find surrogate bugs. Not because it is wrong — because the
# default alphabet is scoped to what UTF-8 can encode.
@given(st.text())
def test_round_trips_utf8_safe_text(value):
    assert url_decode(url_encode(value)) == value


# Can. st.characters() with no arguments includes surrogates, so this
# reaches the class of input the property was written for.
@given(st.text(st.characters()))
def test_round_trips_any_unicode_scalar(value):
    assert url_decode(url_encode(value)) == value

Now the important part, which is not a Hypothesis question at all: which of those two tests do you want? The second will find inputs your system genuinely cannot store, and it will fail. Whether that failure is a bug or an out-of-contract input is a decision about your API, and the strategy is where you record the answer. Writing the domain down first is what turns that from an accident into a specification:

# The contract, stated as a strategy: any Unicode scalar the transport can
# carry, up to the field's real limit, excluding the control characters the
# API documents as rejected.
identifiers = st.text(
    alphabet=st.characters(
        codec="utf-8",
        exclude_categories=("Cc", "Cs"),
    ),
    min_size=1,
    max_size=320,
)

@given(identifiers)
def test_identifier_round_trips(value):
    assert url_decode(url_encode(value)) == value

Anyone reading that knows what the function promises. Anyone reading st.text() knows only that someone had a string-typed parameter.

Constrain in the strategy, not in the body

The reflex when generated data is wrong for a test is to reject it inside the test. Hypothesis supports that: assume(condition) is documented as being "like an assert that marks the test case as bad, rather than failing the test," and it lets Hypothesis "try to avoid similar test cases in future."

It is still the wrong tool for anything a strategy can express, because every rejected input is generation work thrown away:

# Wrong. Roughly half of every generated integer is discarded, and the
# non-empty constraint discards more. Hypothesis is doing work to build
# inputs this test will never look at.
@given(st.integers(), st.lists(st.integers()))
def test_scaling_preserves_length(factor, xs):
    assume(factor > 0)
    assume(len(xs) > 0)
    assert len(scale(xs, factor)) == len(xs)


# Right. The domain is in the strategy, so every generated input is used.
@given(st.integers(min_value=1), st.lists(st.integers(), min_size=1))
def test_scaling_preserves_length(factor, xs):
    assert len(scale(xs, factor)) == len(xs)

Hypothesis will tell you when you have crossed the line, via the filter_too_much health check, documented as a "check for when the test is filtering out too many test cases, either through use of assume() or .filter(), or occasionally for Hypothesis internal reasons." Health check names can be given as strings or as enum members, so suppress_health_check=["filter_too_much"] and suppress_health_check=[HealthCheck.filter_too_much] are equivalent, which is worth knowing so you recognise both in a config file.

Suppressing it is sometimes correct. The docs are relaxed about this, noting that health checks are "proactive warnings, not correctness errors" and encouraging suppression "where you have evaluated they will not pose a problem." The trap is suppressing it without looking, because the reason it fires is that your test is exploring a fraction of the space you think it is.

The measurement is one flag. --hypothesis-show-statistics reports where the generation budget went, and the filter cost appears as named events:

test_even_integers:

  - during generate phase (0.09 seconds):
      - Typical runtimes: < 1ms, ~ 59% in data generation
      - 100 passing, 0 failing, and 32 invalid test cases
      - Events:
        * 54.55%, Retried draw from integers().filter(lambda x: x % 2 == 0) to satisfy filter
        * 24.24%, Aborted test because unable to satisfy integers().filter(lambda x: x % 2 == 0)
  - Stopped because settings.max_examples=100

Read the invalid-case count against the passing count. Thirty-two invalid cases out of a hundred passing is a third of your budget spent generating data that was thrown away, and "aborted because unable to satisfy" means some attempts produced nothing at all.

You can add your own labels to that report with event(), which records a value whose frequency is summarised at the end. It is the fastest way to answer "is this strategy actually producing the interesting shape?" Tag the branch you care about and read the percentage.

Fixtures reset once per test, not once per input

Most Hypothesis health checks are about speed. Two are about correctness, and one of those catches an assumption almost everyone makes.

From the reference: HealthCheck.function_scoped_fixture "indicates that a function-scoped pytest fixture is used by an @given test. Many Hypothesis users expect function-scoped fixtures to reset once per input, but they actually reset once per test. We proactively raise HealthCheck.function_scoped_fixture to ensure you have considered this case." The docs are explicit that this and differing_executors are the exceptions: "with the exception of HealthCheck.function_scoped_fixture and HealthCheck.differing_executors, all health checks warn about performance problems, not correctness errors."

Hypothesis generates a hundred inputs by default, which the too_slow check describes as generating "100 (by default) inputs per test execution", and a function-scoped fixture is set up once for all hundred:

# Wrong. `db` is created once, then a hundred customers are inserted into
# it. Input 87 runs against a table holding 86 earlier customers, so a
# uniqueness bug passes and a pagination bug fails for the wrong reason.
@given(st.emails())
def test_customer_can_be_created(db, email):
    customer = create_customer(db, email)
    assert customer.email == email

# Right. Generate the data, and let the test own its own state inside the
# body so each input starts from the same place.
@given(st.lists(st.emails(), min_size=1, max_size=25, unique=True))
def test_customers_are_unique_by_email(db_factory, emails):
    with db_factory() as db:
        for email in emails:
            create_customer(db, email)
        assert {c.email for c in all_customers(db)} == set(emails)

Notice the second version also changed the property. Once you accept that all hundred inputs share one database, the honest thing to test is a claim about a batch. A list-valued strategy states that directly, with unique=True and a max_size that keeps each input affordable. Fighting the fixture lifetime usually means the property was about the wrong unit.

Pin every counterexample you are given

Shrinking is the part of Hypothesis that earns its cost: it reduces a failing input to a minimal one, so the counterexample you read is the smallest thing that breaks. Throwing that away after fixing the bug is the most common waste in property-based testing, because the next regression will have to be rediscovered by chance.

@example is the documented way to keep it. Hypothesis "will always try [explicit inputs] before generating random inputs," and three of its properties matter here: explicit examples "do not count towards settings.max_examples", they "do not shrink", and "if an explicit example fails, Hypothesis will stop and report the failure without generating any random inputs." So a pinned counterexample costs nothing from your generation budget and fails fast and identically every time, which is exactly what you want from a regression test.

The docs name this use directly: if Hypothesis reports that f(n=[0, math.nan]) fails, "you can add @example(n=[0, math.nan]) to your test to quickly reproduce that failure."

@example("\ud800").via("regression test for AUT-512")
@example("").via("regression test for AUT-377")
@given(st.text(st.characters()))
def test_round_trips_any_unicode_scalar(value):
    assert url_decode(url_encode(value)) == value

.via() is there so the provenance survives: the docs describe it as documenting where an example came from, and note it is also used by tooling that adds or removes @example decorators automatically. There is a matching .xfail() for a counterexample you have accepted and not yet fixed, which takes a reason, a raises tuple, and a condition: @example(...).xfail(raises=ZeroDivisionError) keeps a known break visible instead of deleted.

Two other reproduction routes exist and neither replaces @example. @reproduce_failure(version, blob) replays one exact case from a serialised blob; Hypothesis prints it when settings.print_blob is true, "which is the default in CI". It is deliberately disposable: "intended to be temporarily added to your test suite in order to reproduce a failure … not intended to be a permanent addition", and it "will error if used on a different Hypothesis version than it was created for." Paste it into your editor, not into a commit.

@seed fixes the randomness so a run repeats, but only "assuming that there are no other sources of nondeterminism, such as timing, hash randomization, or external state," and the docs point out that Hypothesis "will only print the seed which would reproduce a failure if a test fails in an unexpected way, for instance inside Hypothesis internals." So the seed is not your normal reproduction path: the example database is, and it is what makes a failing case re-run automatically until it passes. There is a --hypothesis-seed flag for the pytest plugin when you do want it, and setting a seed overrides settings.derandomize, which exists "to enable deterministic CI tests rather than reproducing observed failures."

Last piece of hygiene: attach context with note() rather than print(). A noted value "is reported for the minimal failing test case", so it appears next to the shrunk input rather than a hundred times in captured output.

When uniform randomness is the wrong search

Some bugs are not reachable by chance because the interesting region of the input space is small. target() turns generation into a search: called with an int or float observation, it "gives it feedback with which to guide our search for inputs that will cause an error, in addition to all the usual heuristics." Hypothesis maximises the value, and "almost any metric will work so long as it makes sense to increase it": the docs offer -abs(error) as the idiom for driving a value toward zero, along with queue length, runtime and compression ratio as example metrics.

@given(st.floats(min_value=0, max_value=1e6, allow_nan=False))
def test_interest_accrual_stays_within_a_cent(principal):
    fast = accrue_fast(principal)
    exact = accrue_decimal(principal)
    error = abs(fast - exact)
    target(error, label="accrual error")   # search toward the worst case
    assert error < Decimal("0.01")

The honest limits come from the same page, and they are worth quoting rather than softening: "this is not always a good idea — for example calculating the search metric might take time better spent running more uniformly-random test cases, or your target metric might accidentally lead Hypothesis away from bugs." A metric that rewards long inputs will find long inputs and stop finding the empty one.

So this is the last thing to add, not the first. Reach for it when there is "a natural metric like 'floating-point error', 'load factor' or 'queue length'", as the docs put it, and leave it out when you would have to invent one.

Apply this now

Open your existing property tests and grep the assertions for isinstance, is not None, and comparisons against a literal. Each one is a test whose input is decorative. Replace it with a round-trip, a metamorphic relation, or a comparison against a reference implementation, and if none of the three applies, delete the test rather than leaving something that cannot fail.

Then run the suite once with --hypothesis-show-statistics and read the invalid-case counts. Any test discarding a meaningful share of its budget has its domain in the wrong place; move the constraint from assume() into strategy arguments.

What you are aiming at is a test file where every @given is preceded by the @example decorators recording every counterexample the suite has ever found, each with a .via() naming the ticket. That block is the accumulated memory of the property, and it is the part that stops the same bug arriving twice.

Questions about writing properties

How many inputs is enough?

A hundred is the default, and raising it has sharply diminishing returns compared with widening the domain. If a test has run ten thousand inputs without failing, the likely explanation is that the strategy cannot reach the failing region, not that you need twenty thousand. Spend the effort on the alphabet, the size bounds and the shape of the generated object instead. The statistics output reports how much of the runtime went into generation, which tells you whether more inputs are even affordable.

Should property tests replace example-based tests?

No, and the two answer different questions. An example test pins a specific documented behaviour that a reader can check against the spec: encode("a b") == "a%20b" is worth keeping precisely because it is concrete. A property states what must hold across the domain. The useful arrangement is both in one place: @example decorators for the documented and historically broken cases, @given for the domain, one assertion that covers all of them.

What if the property is slower than the function under test?

That is normal for differential testing, where the oracle is a slow reference implementation, and it is usually still worth it. You are buying correctness evidence, not throughput. Keep such tests out of the fast feedback loop rather than weakening them: a smaller max_examples locally and a larger one on a nightly profile is the standard split, and Hypothesis supports registering named profiles for exactly this.

Is it cheating to shrink the domain until the test passes?

Only if you do it silently. Narrowing the strategy is how you record a contract, and a strategy that excludes control characters because the API rejects them is documentation. What makes it cheating is narrowing it in response to a failure without deciding whether the failure was a bug, at which point the commit that shrinks the domain is the commit that hides the defect. Write the reason in the strategy definition, not in the commit message, where the next reader will not look.

Can Hypothesis test stateful systems?

Yes, through rule-based state machines, which generate sequences of operations rather than single values, and that is the right tool once your invariant is about a system rather than a function, such as "the balance never goes negative across any sequence of deposits and withdrawals". The signal that you need it is a property test whose body is building up state in a loop before asserting, which is the shape the fixture section above arrives at. Note that @seed applies to a state machine class as well as a test function, so the same reproduction story holds.

Primary references

• Hypothesis — Strategies: text(). That the default alphabet is characters(codec='utf-8'), that it "can generate the full unicode range but excludes surrogate characters because they are invalid in the UTF-8 encoding", and that you "can use characters() without arguments to find surrogate-related bugs such as bpo-34454".
• Hypothesis — API: Explicit inputs (@example). That explicit inputs are always tried before random ones; that they "do not count towards settings.max_examples"; that they "do not shrink" and Hypothesis "will stop and report the failure without generating any random inputs" when one fails; the documented use of pinning a reported counterexample; and the .xfail() and .via() modifiers, including that .via supports tooling that adds or removes @example decorators.
• Hypothesis — API: HealthCheck. That health checks are "a proactive warning, not an error"; that filter_too_much checks "for when the test is filtering out too many test cases, either through use of assume() or .filter()"; that too_slow exists because "Hypothesis generates 100 (by default) inputs per test execution"; and that names may be given as strings or enum members in suppress_health_check.
• Hypothesis — API: Correctness health checks. That function_scoped_fixture and differing_executors are the only health checks reporting correctness rather than performance, and the statement that "many Hypothesis users expect function-scoped fixtures to reset once per input, but they actually reset once per test".
• Hypothesis — API: assume(). That assume "is like an assert that marks the test case as bad, rather than failing the test", and that it lets Hypothesis "try to avoid similar test cases in future".
• Hypothesis — API: event(). That events are summarised as frequencies at the end of a run, the worked --hypothesis-show-statistics output showing passing, failing and invalid test-case counts alongside "Retried draw … to satisfy filter" and "Aborted test because unable to satisfy …" percentages, and that two events are the same if their string forms match.
• Hypothesis — API: note(). That a noted value "is reported for the minimal failing test case, and on Verbosity.verbose or higher", which is why it belongs where a print() would otherwise go.
• Hypothesis — API: Reproducing inputs. That @reproduce_failure replays exactly one case from a serialised blob, is printed when print_blob is true ("the default in CI"), is "not intended to be a permanent addition" and "will error if used on a different Hypothesis version"; and that @seed reproduces a run only absent other nondeterminism, overrides settings.derandomize, is exposed as --hypothesis-seed, and is only printed automatically when a test "fails in an unexpected way".
• Hypothesis — API: target(). That target takes a finite int or float observation to "guide our search for inputs that will cause an error, in addition to all the usual heuristics", that Hypothesis maximises it, that "almost any metric will work so long as it makes sense to increase it", and the -abs(error) idiom.
• Hypothesis — API: Targeted property-based testing. The stated trade-off: that targeting "is not always a good idea — for example calculating the search metric might take time better spent running more uniformly-random test cases, or your target metric might accidentally lead Hypothesis away from bugs", and the recommendation to try it where a natural metric such as floating-point error, load factor or queue length exists.
• Hypothesis — API: settings.max_examples. That once this many satisfying test cases have been considered without a failure, "Hypothesis will stop looking", and the note that what the setting name calls "examples" are now referred to as test cases throughout the documentation.