Build Deterministic Python API Test Data Factories

Three failures, all from the same cause and all attributed to something else:

psycopg.errors.UniqueViolation: duplicate key value violates unique
constraint "customers_email_key"

AssertionError: assert 'Margaret Boehm' == 'Danielle Hartman'

FAILED test_order_totals — 41-line payload diff, 2 fields relevant

The first gets blamed on parallelism, the second on a flaky dependency, the third on nothing at all because nobody reads a 41-line diff twice. All three come from generating data without deciding what about it is fixed, what varies, and what the test is actually asserting.

The faker fixture hands every test the same data

Start here, because it inverts what most people assume. Faker's pytest fixture is documented as returning "a session-scoped Faker instance to be used across all tests in your test suite. This instance defaults to the en-US locale, it is reseeded using a seed value of 0 prior to each test, and the .unique remembered generated values are cleared."

Read that twice. The reseed happens before each test, which means the first faker.email() in every test returns the same string. Determinism is already the default; what you do not have is uniqueness across tests.

# Both tests get the same email, because both start from seed 0.
# Run them against a shared database and the second one violates the
# unique index — with an error that says nothing about Faker.
def test_customer_can_register(db, faker):
    register(db, email=faker.email())

def test_customer_can_be_invited(db, faker):
    invite(db, email=faker.email())

The two knobs are session-scoped autouse fixtures. faker_seed changes the seed and faker_session_locale changes the locale. The docs show both, and the locale fixture takes a list so multiple-locale mode is a matter of returning more than one:

# conftest.py
@pytest.fixture(scope="session", autouse=True)
def faker_seed():
    return 12345

@pytest.fixture(scope="session", autouse=True)
def faker_session_locale():
    return ["it_IT", "ja_JP", "en_US"]

Changing the seed does not fix the collision, though. It only moves which identical value every test receives. The collision is structural, and the fix is in the next two sections.

One more distinction to have straight, because it decides whether another test can perturb your factory. Faker.seed() is a class method that "seeds the shared random number generator", and calling it on an instance is an error: Faker raises TypeError: Calling `.seed()` on instances is deprecated. Use the class method `Faker.seed()` instead. The project explains the change as dealing with "a non-explicit legacy behavior involving a shared random.Random instance." The per-instance alternative is seed_instance(), which switches a generator "to use its own instance of random.Random, separated from the shared one".

from faker import Faker

# Shared: any other test or library calling Faker.seed() moves this.
Faker.seed(4321)

# Isolated: this generator has its own random.Random and cannot be
# perturbed by anything else in the process. This is what a factory wants.
fake = Faker()
fake.seed_instance(4321)

Generated values are inputs, never expected outputs

The second failure in the opening, assert 'Margaret Boehm' == 'Danielle Hartman', is a test that recorded a seeded value as its expectation. Faker is unambiguous that this is not supported: "as we keep updating datasets, results are not guaranteed to be consistent across patch versions. If you hardcode results in your test, make sure you pinned the version of Faker down to the patch number."

A seed reproduces a run, given the same version. It is not a stable contract, and pinning Faker to a patch release to protect a hard-coded name is trading a real dependency constraint for a test that was asserting the wrong thing anyway.

# Wrong. The expectation is a fact about Faker's dataset, not about the
# system under test. A patch bump breaks it, and fixing it teaches nobody
# anything.
def test_customer_name_is_stored(db, faker):
    Faker.seed(4321)
    create_customer(db, name=faker.name())
    assert fetch_customer(db).name == "Margaret Boehm"


# Right. The assertion is a relationship — what went in comes back out —
# so it holds for any name the factory produces.
def test_customer_name_is_stored(db, faker):
    name = faker.name()
    create_customer(db, name=name)
    assert fetch_customer(db).name == name

The rule generalises past Faker: any value your test did not choose deliberately must not appear on the right-hand side of an assertion. If a test needs a specific value (a name with a combining character, an email at exactly the length limit), that value is part of the test, so write it in the test.

Derive identity rather than drawing it

Uniqueness and reproducibility look like opposites and are not. You get both by computing identity from something already unique (the test's own name) instead of sampling it.

uuid5 is the tool. Python documents it as generating "a UUID based on the SHA-1 hash of a namespace identifier (which is a UUID) and a name (which is a bytes object or a string that will be encoded using UTF-8)", per RFC 9562 §5.5. Same inputs give the same UUID; different inputs give a different one. No randomness, so nothing to seed and nothing to record.

import uuid

# NAMESPACE_URL is one of the module's predefined namespaces, documented
# for names that are URLs — so a synthetic URL keyed to the test reads
# correctly and stays inside the intended use.
def derived_id(request, kind: str, index: int = 0) -> uuid.UUID:
    name = f"https://tests.invalid/{request.node.nodeid}/{kind}/{index}"
    return uuid.uuid5(uuid.NAMESPACE_URL, name)


@pytest.fixture
def order_factory(request, faker):
    counter = itertools.count()

    def make(**overrides):
        index = next(counter)
        oid = derived_id(request, "order", index)
        base = {
            "id": str(oid),
            # The idempotency key is derived too, so a retry inside the
            # test reuses it and a different test cannot collide with it.
            "idempotency_key": str(derived_id(request, "idem", index)),
            "customer_email": f"c-{oid.hex[:12]}@tests.invalid",
            "currency": "INR",
            "items": [{"sku": "SKU-1", "qty": 1, "unit_price_minor": 19900}],
        }
        return {**base, **overrides}

    return make

Three properties come out of this for free. The email is unique across tests because the node id is. It is identical on every run of the same test, so a failure is reproducible without recording a seed. And it is stable under pytest-xdist, because the node id does not depend on which worker picked the test up.

Faker's .unique proxy solves a narrower problem and it is worth knowing its limits before reaching for it. It "guarantee[s] that any generated values are unique for this specific instance", and fake.unique.clear() resets the memory, which the pytest fixture already does before every test, so its guarantee is per-test only. It also fails loudly when the pool is too small: "to avoid infinite loops, after a number of attempts to find a unique value, Faker will throw a UniquenessException", and the docs add the warning that matters at scale: "beware of the birthday paradox, collisions are more likely than you'd think." fake.unique.boolean() raises on the third call, because there are only two booleans. Finally, "only hashable arguments and return values can be used with .unique".

So: .unique for display fields that must differ within one test, derived UUIDs for anything the system treats as an identity.

One valid baseline plus overrides that must be real

The third opening failure, the 41-line diff, is a factory that builds a maximal payload. Every field it sets is a field a reader has to rule out. The fix is a baseline that is the smallest payload the API accepts, and named scenarios on top of it.

The pitfall that makes factories untrustworthy is silently swallowing overrides that do not exist. {**base, **overrides} above happily accepts currancy="USD" and gives you a payload with both keys, so the test passes while testing nothing. Give the baseline a real type and the language raises for you:

from dataclasses import dataclass, replace, asdict

@dataclass(frozen=True, slots=True)
class Order:
    id: str
    idempotency_key: str
    customer_email: str
    currency: str = "INR"
    items: tuple[dict, ...] = ()

def make(request, **overrides) -> Order:
    base = Order(**baseline_fields(request))
    # replace() raises TypeError on a field Order does not declare, so a
    # misspelled override fails at the call site instead of vanishing.
    return replace(base, **overrides)

Then express scenarios as functions rather than as flags, because a flag tells the reader nothing about what it changes:

# Each name states the rule the test is about. The override list is the
# diff a reviewer needs to read, and it is two lines rather than forty.
def order_over_vat_threshold(request):
    return make(request, currency="EUR",
                items=({"sku": "S", "qty": 1, "unit_price_minor": 250_00},))

def order_with_exempt_plan(request):
    return make(request, items=({"sku": "EDU-1", "qty": 1,
                                 "unit_price_minor": 100_00},))

The cost is a longer factory module and a name to invent per scenario. What you get back is that a failing test's payload is legible without a debugger, and that adding a required field to the API breaks the baseline once rather than in every test.

Make the factory check its own output

A factory drifts. The API adds a required field, tightens a pattern, or narrows an enum, and the factory keeps producing payloads that were valid last quarter. Validating inside the factory turns that into one failure with a clear message instead of a hundred confusing ones.

Two details from jsonschema make the difference between a real check and a decorative one.

First, use iter_errors rather than validate. validate(instance) "raises jsonschema.exceptions.ValidationError if the instance is invalid." So it reports the first problem and stops. iter_errors(instance) will "lazily yield each of the validation errors in the given instance", which is what you want when a stale factory has three fields wrong:

from jsonschema.validators import Draft202012Validator

def validated(payload: dict, schema: dict) -> dict:
    validator = Draft202012Validator(
        schema,
        # Without this, "format" is not checked at all.
        format_checker=Draft202012Validator.FORMAT_CHECKER,
    )
    errors = sorted(validator.iter_errors(payload), key=str)
    if errors:
        raise AssertionError(
            "factory produced an invalid payload:\n"
            + "\n".join(f"  {e.json_path}: {e.message}" for e in errors)
        )
    return payload

Second, and this is the one that quietly hollows out schema checks: the format keyword does nothing by default. The docs are explicit: "by default, as per the specification, no validation is enforced", and "optionally however, validation can be enabled by hooking a format-checking object into a Validator." So a schema declaring {"format": "email"} accepts "not-an-email" unless you pass format_checker. Some checks also need extras installed, via pip install jsonschema[format] or the GPL-free jsonschema[format-nongpl].

A factory that generates emails and validates them against an email format with the checker disabled is a factory testing nothing about its emails. That is worth a one-line assertion in your own test suite: feed the factory's schema a known-bad value and confirm the validator rejects it.

Apply this now

Grep the test tree for faker. and fake. outside your factory module. Every hit is generation logic living in a test, which is where it becomes invisible in a diff. Move them behind a factory call with named overrides.

Then grep the assertions for any comparison whose right-hand side came from a generator. Those are the tests that will break on a Faker upgrade for reasons unrelated to your code. Rewrite each as an echo-back or a relationship.

A failing test should carry three things in its own output: the factory scenario by name, the resolved payload as the factory produced it, and the validation result. Log those at the point the factory returns, not from the test, so every test gets them without asking. Once that is in place, the reproduction instruction for any failure is the node id: no seed to copy, because identity was derived rather than drawn.

Questions about test data factories

Should the factory write to the database or return a payload?

Return the payload, and let a separate fixture persist it. A factory that inserts is doing two jobs, and the moment you need the same payload for an API call rather than an insert you have to duplicate it. Splitting them also keeps the factory usable in tests that assert on validation errors, where nothing should be persisted at all.

Is Faker worth using if values must not be asserted on?

Yes, for two things: producing shapes you would not have thought to type, and making payloads look like production data so that a length or encoding assumption surfaces. What it is not for is identity or expected values. A reasonable division is Faker for the descriptive fields, derived UUIDs for anything with a uniqueness constraint, and hand-written literals for any value the assertion depends on.

How do I keep factories in step with a changing API?

Point the factory's validation at the same schema artefact the API publishes rather than a copy. If the schema lives in the service repository, vendor it as a build step so a stale copy fails loudly on the next pull. The failure mode you are designing against is not the API changing. It is the API changing while the factory's private copy of the schema does not.

Do derived UUIDs break when a test is renamed?

They change, which is the correct behaviour and worth understanding rather than working around. The identity is a function of the test's name, so renaming the test produces new data, which is what you want, because a renamed test is being re-run from a clean state. What must not happen is deriving from something that changes between runs of the same test, such as a timestamp or the worker id.

What about parametrized tests sharing a node id prefix?

They do not collide, because a parametrized node id includes its case identifier in brackets, which is one more reason to give parametrized cases explicit IDs rather than letting them be generated. A derived UUID built from a node id ending in [a0-b0] is unique but tells you nothing; one ending in [eu-above-threshold] is unique and legible in a database row.

Primary references

• Faker — Pytest Fixtures. That the faker fixture returns a session-scoped instance defaulting to en-US, that "it is reseeded using a seed value of 0 prior to each test", and that "the .unique remembered generated values are cleared" at the same point. Also the session-scoped autouse faker_seed and faker_session_locale override fixtures, including the list form for multiple locales, and that the fixture is function-scoped and configurable despite the shared instance.
• Faker — Seeding the Generator. That Faker.seed() "seeds the shared random number generator" and "a Seed produces the same result when the same methods with the same version of faker are called"; that seed_instance() switches a generator "to use its own instance of random.Random, separated from the shared one"; and the warning that "as we keep updating datasets, results are not guaranteed to be consistent across patch versions", so hard-coded results require pinning Faker to a patch release.
• Faker — Unique values. That .unique guarantees uniqueness "for this specific instance"; that fake.unique.clear() clears the seen values; that Faker "will throw a UniquenessException" after a number of failed attempts, with the explicit "beware of the birthday paradox" caution and the fake.unique.boolean() example that raises on the third call; and that "only hashable arguments and return values can be used with .unique".
• Faker — Faker class: breaking change and upgrade guide. The TypeError quoted earlier in this article, raised when seeding is attempted through an instance rather than the class, together with the stated rationale of resolving "a non-explicit legacy behavior involving a shared random.Random instance".
• Python — uuid.uuid5. That uuid5 generates "a UUID based on the SHA-1 hash of a namespace identifier (which is a UUID) and a name (which is a bytes object or a string that will be encoded using UTF-8) according to RFC 9562, §5.5." The property that makes derived identity both unique and reproducible without a recorded seed.
• Python — uuid.NAMESPACE_URL. That the module defines namespace identifiers for use with uuid3() or uuid5(), and that with NAMESPACE_URL "the name string is a URL", which is why a synthetic URL keyed to the test node id is the right shape for the name argument.
• jsonschema — The Validator Protocol. That iter_errors(instance) will "lazily yield each of the validation errors in the given instance", with the worked example producing two messages from one instance, while validate(instance) "raises jsonschema.exceptions.ValidationError if the instance is invalid" and therefore stops at the first problem.
• jsonschema — Validating Formats. That for the format keyword, "by default, as per the specification, no validation is enforced", and that it must be switched on "by hooking a format-checking object into a Validator" such as Draft202012Validator.FORMAT_CHECKER; plus that some formats need the jsonschema[format] or jsonschema[format-nongpl] extras installed.
• Python — random.Random. The class behind both Faker seeding modes: an instantiable generator whose state is independent of the module-level shared instance, which is what seed_instance() switches a Faker generator over to.