# Run pytest in Parallel Without Shared-State Flakes

> Adding workers does not create shared state — it reveals that the suite only ever worked because it ran one test at a time. This guide starts from the fact that every xdist worker collects independently and runs its own copy of each session-scoped fixture, then covers deriving a reclaimable namespace from worker_id, one PostgreSQL schema per worker, why collection order has to agree across workers, and why -s cannot work under xdist at all.

- Author: [Shashank Rawlani](https://shashank.rawlani.com)

- Published: 2026-09-21T13:30:00.000Z

- Updated: 2026-09-07T11:52:06.356Z

- Category: Automation Tutorials

- Canonical source: https://automationtester.in/blog/automation-tutorials/pytest-parallel-workers-shared-state-isolation

- Tags: pytest, python, pytest-xdist, parallel-testing, flaky-tests, test-isolation

**Quick answer:** pytest-xdist gives you process isolation and nothing else. Each worker collects independently and runs its own copy of every session-scoped fixture, so "runs once per session" becomes "runs once per worker" the moment you pass `-n`. Partition every mutable external resource by a namespace *derived* from `worker_id` rather than randomly generated, and guard anything that genuinely must happen once with a file lock — xdist has no built-in mechanism for it.

`-n auto` takes a 22-minute suite to six minutes, and then roughly one run in four fails. The failures move around, and they look like this:

```
psycopg.errors.UniqueViolation: duplicate key value violates unique
constraint "users_email_key"

OSError: [Errno 48] Address already in use

FileNotFoundError: [Errno 2] No such file or directory:
'.artifacts/response-dump.json'
```

None of these is a concurrency bug in your application. All three are the same thing: a resource that lived outside the process, being written by four processes that each believed they owned it. Adding workers did not create the sharing — it revealed that the suite had only ever worked because it ran one test at a time.

## Session scope becomes once per worker

This is the fact that invalidates most existing conftest files, and it is stated directly in xdist's own how-to: "pytest-xdist is designed so that each worker process will perform its own collection and execute a subset of all tests. This means that tests in different processes requesting a high-level scoped fixture (for example session) will execute the fixture code more than once, which breaks expectations and might be undesired in certain situations."

So the session fixture that applies migrations applies them four times. The one that seeds reference data seeds it four times, racing. The one that starts a container starts four containers.

xdist is explicit that it does not solve this for you: it "does not have a builtin support for ensuring a session-scoped fixture is executed exactly once," and the documented approach is a lock file for inter-process communication. The pattern is worth copying exactly, because the part people get wrong is which directory to put the lock in:

```
import json
import pytest
from filelock import FileLock

@pytest.fixture(scope="session")
def session_data(tmp_path_factory, worker_id):
    if worker_id == "master":
        # Not running with multiple workers: produce the data and let
        # pytest's own fixture caching do the rest.
        return produce_expensive_data()

    # getbasetemp() is per-worker. Its parent is the directory every
    # worker shares, which is the only place a lock is visible to all.
    root_tmp_dir = tmp_path_factory.getbasetemp().parent

    fn = root_tmp_dir / "data.json"
    with FileLock(str(fn) + ".lock"):
        if fn.is_file():
            data = json.loads(fn.read_text())
        else:
            data = produce_expensive_data()
            fn.write_text(json.dumps(data))
    return data
```

The `.parent` is the whole trick. Each worker gets its own base temp directory — which is why `tmp_path` needs no partitioning of its own — and the shared parent is what makes a cross-process lock possible without inventing a path. The `worker_id == "master"` branch exists because the fixture must still work when someone runs a single file without `-n`.

xdist names database setup as a case this applies to, describing the same technique as suitable for work that "needs to execute exactly once per test session, like initializing a database service and populating initial tables."

## Derive the namespace instead of generating one

Once you accept that each worker needs its own copy of the mutable resources, the question is what to call them. There are two options and they are not close.

```
# Wrong. Unique, and unusable. Nothing can find these afterwards, so a
# crashed run leaves schemas nobody will ever drop, and the count grows
# every CI build until someone notices the disk.
@pytest.fixture(scope="session")
def schema(connection):
    name = f"test_{uuid.uuid4().hex[:8]}"
    connection.execute(f"CREATE SCHEMA {name}")
    yield name
    connection.execute(f"DROP SCHEMA {name} CASCADE")
```

A derived name is reclaimable. xdist provides the input as a fixture: `worker_id`, which returns names like `gw0` and `gw2`, and returns `"master"` when xdist is disabled — the docs give exactly this use as the example, "use a different account in each xdist worker."

The same value is available three other ways, and each is right in a different place. Worker processes carry `PYTEST_XDIST_WORKER` (the name, e.g. `gw2`) and `PYTEST_XDIST_WORKER_COUNT` (the total, e.g. `4`) in their environment, which is how you reach it from `pytest_configure` or from a subprocess your tests launch. Since xdist 2.0 there are also functions — `xdist.get_xdist_worker_id()` returns "the id of the current worker ('gw0', 'gw1', etc) or 'master' if running on the controller node", and `xdist.is_xdist_worker()` answers the boolean, both accepting a request or session object. And the worker id is recorded on the report: the docs note it "is stored in the TestReport as well, under the `worker_id` attribute," which is what lets a report consumer attribute a failure to a worker after the fact.

For uniqueness across concurrent *runs* rather than across workers, there is `testrun_uid`. Its documented purpose is to "globally distinguish one test run from others in your workers," and xdist's own example combines it with a POSIX semaphore to create a per-run database exactly once:

```
@pytest.fixture(scope="session", autouse=True)
def create_unique_database(testrun_uid):
    database_url = f"psql://myapp-{testrun_uid}"

    with Semaphore(f"/{testrun_uid}-lock", flags=O_CREAT, initial_value=1):
        if not database_exists(database_url):
            create_database(database_url)
```

Note what the semaphore is for. Every worker runs this fixture, so without the lock they race to create the same database. That is the same problem as the previous section wearing different clothes, and it is the shape of every "once per run" operation under xdist.

Use both axes together and the naming stops being a decision: `testrun_uid` separates concurrent runs, `worker_id` separates workers inside a run.

## One schema per worker beats one database per worker

For PostgreSQL the cheap partition is a schema, not a database, because switching between them costs a `SET` rather than a new connection and a new pool.

What makes it work is the search path. PostgreSQL's documentation states that "the first schema named in the search path is called the current schema," and that "aside from being the first schema searched, it is also the schema in which new tables will be created if the `CREATE TABLE` command does not specify a schema name." So an unqualified query in a worker whose search path starts at its own schema never touches another worker's rows, and no application code has to know it is running under a test.

```
@pytest.fixture(scope="session")
def worker_schema(engine, worker_id, testrun_uid):
    # Derived, so a crashed run leaves a name a later run can find and
    # drop. Both axes: run, then worker.
    name = f"t_{testrun_uid[:8]}_{worker_id}"

    with engine.begin() as conn:
        conn.exec_driver_sql(f'DROP SCHEMA IF EXISTS "{name}" CASCADE')
        conn.exec_driver_sql(f'CREATE SCHEMA "{name}"')
        apply_migrations(conn, schema=name)

    yield name

    with engine.begin() as conn:
        conn.exec_driver_sql(f'DROP SCHEMA IF EXISTS "{name}" CASCADE')


@pytest.fixture
def db(engine, worker_schema):
    conn = engine.connect()
    # Unqualified DDL and DML now resolve inside this worker's schema.
    conn.exec_driver_sql(f'SET search_path TO "{worker_schema}", public')
    transaction = conn.begin()
    yield Session(bind=conn)
    transaction.rollback()
    conn.close()
```

The cost is honest: migrations now run once per worker rather than once per run, which on a large schema can add ten seconds times the worker count to startup. If that dominates, this is precisely the operation to move behind the file lock from the previous section — migrate a template schema once, then have each worker copy it — and the trade you are making is startup time against a more complicated conftest.

Keep the per-test isolation separate from the per-worker isolation. The schema stops workers colliding; the transaction rollback stops tests within a worker colliding. Neither substitutes for the other, and a suite that has only one of them fails in a way that looks like the other's absence.

## Ports, artifacts, and anything with a fixed name

Everything with a name your code chose is a collision. Ports and file paths are the two that produce the most confusing failures, because both fail at the OS level with a message that says nothing about tests.

A random port is a race between choosing and binding — the number is free when you check and taken by the time you use it, and the window widens with worker count. Deriving the port removes the choice entirely:

```
@pytest.fixture(scope="session")
def stub_server_port(worker_id):
    index = 0 if worker_id == "master" else int(worker_id.removeprefix("gw"))
    # PORT_BASE from the environment, so two concurrent CI jobs on one
    # host can be given non-overlapping ranges.
    return int(os.environ.get("PORT_BASE", "9400")) + index
```

When the server is one you control rather than one you configure, the better answer is to bind to port 0 and ask the socket what it got — `getsockname()` "return[s] the socket's own address", so the assignment happens once, atomically, and nothing has to be reserved.

Artifacts are the same problem with a worse symptom, because a collision silently overwrites rather than raising. Anything your tests or fixtures write — screenshots, request dumps, downloaded files, HAR captures — needs the worker in its path:

```
@pytest.fixture(scope="session")
def artifact_dir(worker_id):
    path = Path(".artifacts") / worker_id
    path.mkdir(parents=True, exist_ok=True)
    return path
```

Caches, message-queue names and key prefixes follow the same rule. A Redis prefix of `tests:gw2:` and a queue of `orders.test.gw2` cost one f-string each and remove an entire class of intermittent failure.

## Tests that must share a resource belong together

Some tests cannot be partitioned. They exercise a genuinely singular thing — a licence-limited service, a hardware device, a third-party sandbox account with one seat. Partitioning is not available, so the answer is to keep them in one process.

xdist's distribution modes exist for this, and they differ in what unit they keep together. `--dist load` is the default and "sends pending tests to any worker that is available, without any guaranteed order." `--dist loadfile` groups "by their containing file", which "guarantees that all tests in a file run in the same worker." `--dist loadscope` groups "by module for test functions and by class for test methods", with "grouping by class tak[ing] priority over grouping by module" — useful, as the docs say, when you have expensive module- or class-level fixtures.

The precise one is `--dist loadgroup`, which groups by an explicit mark rather than by file layout:

```
@pytest.mark.xdist_group(name="payments-sandbox")
def test_charge_succeeds(): ...

class TestRefunds:
    @pytest.mark.xdist_group("payments-sandbox")
    def test_partial_refund(self): ...
```

Both land on the same worker regardless of which files they live in. The docs add two details worth knowing: if a test carries multiple groups "they will be joined together into a new group, the order of the marks doesn't matter", and the mark "works along with marks from fixtures and from the `pytestmark` global variable" — so a parametrized fixture can assign the group, which is how a per-browser or per-container parameter keeps its tests pinned.

The cost is the one you would expect and it is easy to underestimate: a group runs serially, so the largest group becomes the floor on your wall-clock time. Twelve minutes of tests in one `xdist_group` means the suite cannot finish in under twelve minutes no matter how many cores you buy.

## Collection has to agree across workers

This one has no analogue in serial runs, so it arrives as a parallel-only failure with a message about scheduling. xdist's known limitations are blunt: "It is not possible to have tests that differ in order or their amount across workers."

Because every worker collects independently, any non-determinism in collection produces divergent test lists. The docs give the canonical trigger — parametrizing over a set:

```
# Wrong. Sets are not ordered, so workers can collect these in different
# orders and the run errors rather than failing a test.
@pytest.mark.parametrize("param", {"a", "b"})
def test_pytest_parametrize_unordered(param):
    pass

# Right, either way: convert to a list, or sort.
@pytest.mark.parametrize("param", ["a", "b"])
@pytest.mark.parametrize("param", sorted({"a", "b"}))
```

The documented cause is stated as being "especially true with `pytest.mark.parametrize`, when values are produced with sets or other unordered iterables/generators." In practice the set literal is the rare case; the common ones are a parameter table built from `os.listdir()`, from `glob`, from a dictionary's keys, or from a database query without an `ORDER BY`. Any of those can hand two workers different sequences. Wrap every generated parameter list in `sorted()` — it costs nothing and it is the only property xdist requires of you here.

## Minus-s does not work, so write files

The reflex when a parallel test misbehaves is to run it with `-s` and watch. That reflex does not work here, and the reason is architectural rather than a missing feature. Per the known limitations, `-s`/`--capture=no` "does not work with pytest-xdist because execnet the underlying library used for communication between master and workers, does not support transferring stdout/stderr from workers." The docs add that "currently, there are no plans to support this."

So the evidence has to be written, not watched — and written to per-worker paths. xdist documents the pattern, using the worker name from the environment because `pytest_configure` runs before fixtures exist:

```
# content of conftest.py
def pytest_configure(config):
    worker_id = os.environ.get("PYTEST_XDIST_WORKER")
    if worker_id is not None:
        logging.basicConfig(
            format=config.getini("log_file_format"),
            filename=f"tests_{worker_id}.log",
            level=config.getini("log_file_level"),
        )
```

With `-n3` that produces `tests_gw0.log`, `tests_gw1.log` and `tests_gw2.log`. The `if worker_id is not None` guard matters: without `-n` the variable is absent and you want normal terminal logging back.

Put the namespace values in those logs at session start — worker id, run uid, schema name, port, artifact directory. A parallel failure is diagnosable when you can read which partition it happened in, and almost undiagnosable when you cannot.

## Apply this now

Run the suite twice, once with `-n0` and once with `-n auto`, and diff the failures. Anything that fails only in the second run is a shared resource, and the error message names the resource type — a unique-constraint violation is data, `Address already in use` is a port, a missing or unexpected file is a path.

Then inventory in the other direction: grep the test tree for every fixed name your fixtures produce. Schema and database names, ports, directories under a repository-relative path, Redis prefixes, queue names, and any account or tenant identifier hard-coded in a fixture. Each one needs `worker_id` in it or a documented reason it does not.

Ten consecutive green `-n auto` runs is the bar, alongside a per-worker log whose first line records that worker's namespace. Ten passes is the bar rather than one, because the failures this article is about are probabilistic and a single green run is not information.

## Questions about parallel test isolation

### How many workers should I actually use?

`-n auto` uses "as many processes as your computer has physical CPU cores", and `-n logical` uses logical cores instead — the latter needing `psutil`, falling back to `auto` behaviour if it is missing or cannot determine the count. For I/O-bound suites, logical cores or a hand-picked number above the core count often wins, because workers spend their time waiting. Two knobs are worth knowing when CI and laptops need different answers: the `PYTEST_XDIST_AUTO_NUM_WORKERS` environment variable, and the `pytest_xdist_auto_num_workers(config)` hook, which can inspect `config.option.numprocesses` and return `None` to fall back. If both are set, the hook wins.

### Does a database transaction rollback make workers safe?

No, and this is the most common wrong answer to parallelism. Rollback isolates tests from each other *within* one connection. Two workers holding two connections to the same schema still contend on unique indexes, sequences, advisory locks and anything with a fixed primary key. Rollback and partitioning solve different problems; you need both, and the symptom of having only rollback is exactly the unique-constraint violation at the top of this article.

### What about coverage under parallel workers?

Coverage data files collide for the same reason artifacts do, so the collection has to be per-process and merged afterwards rather than written to one file. Treat it as another resource on your inventory, not as a special case: the question to ask is what filename each worker writes, and whether two workers can pick the same one. If they can, it is the same bug as the artifact directory.

### Should tests run in parallel in CI only?

Running parallel locally and serially in CI is the worst configuration, and the reverse is nearly as bad — whichever mode you use less often is where the isolation bugs accumulate unseen. Pick `-n auto` everywhere and keep `-n0` for debugging a specific failure. The `worker_id == "master"` branch that xdist's own examples use exists so the same conftest supports both, which means you do not have to choose one and maintain a second path.

### Can I see which worker ran a failing test?

Yes — the worker id is on the `TestReport` under its `worker_id` attribute, so a small `pytest_runtest_logreport` hook or a report plugin can surface it. That is worth wiring up early, because "this test fails only on gw3" is a very different investigation from "this test is flaky", and the two are indistinguishable in default output.

## Primary references

- [pytest-xdist — Making session-scoped fixtures execute only once](https://pytest-xdist.readthedocs.io/en/stable/how-to.html#making-session-scoped-fixtures-execute-only-once). That "each worker process will perform its own collection and execute a subset of all tests", so session-scoped fixtures "execute the fixture code more than once"; that xdist "does not have a builtin support" for running one exactly once; and the documented `FileLock` pattern using `tmp_path_factory.getbasetemp().parent` as "the temp directory shared by all workers", including the `worker_id == "master"` branch.

- [pytest-xdist — Identifying the worker process during a test](https://pytest-xdist.readthedocs.io/en/stable/how-to.html#identifying-the-worker-process-during-a-test). The `worker_id` fixture and that it returns `"master"` when xdist is disabled; the `PYTEST_XDIST_WORKER` and `PYTEST_XDIST_WORKER_COUNT` environment variables; that the worker id is stored on the `TestReport` under `worker_id`; and the xdist 2.0 functions `is_xdist_worker`, `is_xdist_controller` and `get_xdist_worker_id`.

- [pytest-xdist — Uniquely identifying the current test run](https://pytest-xdist.readthedocs.io/en/stable/how-to.html#uniquely-identifying-the-current-test-run). The `testrun_uid` fixture, its purpose of globally distinguishing one run from another in the workers, and the worked example that guards per-run database creation with a POSIX semaphore because every worker runs the fixture.

- [pytest-xdist — Known limitations: order and amount of tests must be consistent](https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html#order-and-amount-of-test-must-be-consistent). That "it is not possible to have tests that differ in order or their amount across workers", that this is especially true when parametrize values come from "sets or other unordered iterables/generators", and the two documented workarounds of converting to a list or sorting.

- [pytest-xdist — Known limitations: output from workers](https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html#output-stdout-and-stderr-from-workers). That `-s`/`--capture=no` "does not work with pytest-xdist because execnet … does not support transferring stdout/stderr from workers", and that "currently, there are no plans to support this in pytest-xdist".

- [pytest-xdist — Running tests across multiple CPUs](https://pytest-xdist.readthedocs.io/en/stable/distribution.html#running-tests-across-multiple-cpus). That `-n auto` uses physical cores and `-n logical` logical ones (requiring `psutil`, falling back to auto); that `-n 0` disables xdist; the `PYTEST_XDIST_AUTO_NUM_WORKERS` variable and `pytest_xdist_auto_num_workers` hook with the hook taking priority; and the behaviour of `--dist load`, `loadscope`, `loadfile` and `loadgroup`, including that `xdist_group` marks are joined when a test has several and work alongside fixture marks and `pytestmark`.

- [pytest-xdist — Creating one log file for each worker](https://pytest-xdist.readthedocs.io/en/stable/how-to.html#creating-one-log-file-for-each-worker). The `pytest_configure` pattern that reads `PYTEST_XDIST_WORKER` from the environment to build a per-worker log filename, and that `-n3` therefore produces `tests_gw0.log`, `tests_gw1.log` and `tests_gw2.log`.

- [PostgreSQL — Schemas: the schema search path](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH). That "the first schema named in the search path is called the current schema", and that "aside from being the first schema searched, it is also the schema in which new tables will be created if the CREATE TABLE command does not specify a schema name" — which is what makes a per-worker schema transparent to unqualified application queries.

- [Python — socket: getsockname](https://docs.python.org/3/library/socket.html#socket.socket.getsockname). That `getsockname()` returns "the socket's own address", which is how a server bound to port 0 reports the port it was actually given rather than one your test had to reserve in advance.

## Continue reading on AutomationTester.in

- [Debug pytest-asyncio Event-Loop and Fixture Failures](https://automationtester.in/blog/automation-tutorials/debug-pytest-asyncio-event-loop-fixture-failures)
- [Use pytest Parametrization Without Hiding Test Intent](https://automationtester.in/blog/automation-tutorials/pytest-parametrization-with-clear-test-intent)
- [Design pytest Fixtures as Explicit Resource Contracts](https://automationtester.in/blog/automation-tutorials/pytest-fixtures-explicit-resource-contracts)
- [Detect Breaking API Changes Before Consumers Do](https://automationtester.in/blog/automation-tutorials/detect-breaking-api-changes-before-consumers)
- [Verify Pagination, Filtering, and Sorting Contracts](https://automationtester.in/blog/automation-tutorials/api-pagination-filtering-sorting-contract-tests)

Source: [Run pytest in Parallel Without Shared-State Flakes](https://automationtester.in/blog/automation-tutorials/pytest-parallel-workers-shared-state-isolation) by Shashank Rawlani.