Test API Idempotency Under Retries and Timeouts

The duplicate charge that reaches production is almost never the one an idempotency suite reproduces. The suite sends a request, reads the response, sends it again with the same key, and asserts that the two payment identifiers match. The failure has a different shape: a mobile client gives up at three seconds, the server commits at 3.2, and the retry arrives while the first write is still in flight.

That is a concurrent duplicate, following an outcome the client never observed. The sequential double-submit the suite covers is a different code path, and it is the one that was already working.

The contract you are testing is not in the HTTP specification

Start by being clear about what HTTP does and does not give you here. RFC 9110 defines a method as idempotent when "the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request", and names exactly which ones qualify: PUT, DELETE, and the safe methods. POST is not on that list, and no header makes it so as far as the core specification is concerned.

The Idempotency-Key header everyone reaches for is an IETF Internet-Draft, draft-ietf-httpapi-idempotency-key-header. As of September 2026 its latest revision is 07, published in October 2025, and it expired on 18 April 2026 without being published as an RFC. It is a useful and widely implemented design, and it is not a standard you can hold a vendor to. The draft itself is candid about this: without prior knowledge, a client "cannot assume the server will respect this request".

Two consequences follow for a test suite. The authoritative document is the API's own published idempotency policy, which the draft requires resources to publish, and any expectation you cannot find in that document is a guess. And the draft's own syntax rule is one almost nobody implements: the field is defined as an Item Structured Header whose value MUST be a String, meaning the value is quoted.

POST /v1/payments HTTP/1.1
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"

# What virtually every SDK actually sends:
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324

Worth knowing before you write an assertion about it: the draft normatively references RFC 8941 for structured fields, and RFC 8941 has since been obsoleted by RFC 9651. Test what your service documents, and if it accepts both spellings, write a case for each so a future parser change cannot silently reject every client at once.

Five facts to get in writing before the first test

Every assertion below depends on a decision somebody made and may not have recorded. Get these five out of the API documentation, or out of the engineer who wrote the middleware, before writing test names:

• Key scope. Is a key unique per endpoint, per account, or globally? The same UUID sent to two different operations is either two independent operations or a conflict, and the answer changes what a cross-endpoint test asserts.
• Fingerprint definition. The draft describes an optional fingerprint derived from the payload, computed by the resource, and lists several approaches: a checksum over the whole body, a checksum over selected elements, or a field-by-field match on some subset. Which fields are in it decides whether adding an idempotent-irrelevant field like a client trace ID counts as a payload change.
• Retention window. The draft says the resource SHOULD define an expiration policy and publish it. After the window, the same key is a new operation. A window nobody wrote down is a window nobody can test.
• Concurrent behaviour. Conflict, or block and return the first result?
• Replay of a failure. If the first attempt ended in a declined card, does the replay re-attempt, or return the decline?

That last one is the most commonly unspecified and the most expensive to get wrong. The draft's position is that a retry after the original completed should return "the result of the previously completed operation, success or an error" — a stored 402 stays a 402 rather than becoming a second authorization attempt against the card network.

Making the first attempt genuinely ambiguous

This is the part a sequential double-submit test skips, and it is the only part that reproduces the production failure. You need a first attempt whose outcome the client never learns while the server proceeds to completion.

Playwright's APIRequestContext gained a signal option in version 1.62, which aborts the client side of a request in flight. That is exactly the shape of the failure: the client stops waiting, and nothing about that instructs the server to stop working.

import { test, expect } from '@playwright/test';
import { randomUUID } from 'node:crypto';

test('a payment committed after the client gave up is not charged twice', async ({ request }) => {
  const key = randomUUID();
  const body = { accountId: 'acct_7741', amountMinor: 249900, currency: 'INR' };

  // 1. Abandon the first attempt mid-flight. The server keeps going.
  const controller = new AbortController();
  setTimeout(() => controller.abort(), 120);

  await expect(
    request.post('/v1/payments', {
      headers: { 'Idempotency-Key': key },
      data: body,
      signal: controller.signal,
      timeout: 0,
    }),
  ).rejects.toThrow();

  // 2. Confirm the ambiguity is real. If the server never committed, this test
  //    is exercising a first-time request and proves nothing about replay.
  await expect
    .poll(() => paymentCountForKey(request, key), { timeout: 10_000 })
    .toBe(1);

  // 3. Now replay exactly as a retrying client would.
  const replay = await request.post('/v1/payments', {
    headers: { 'Idempotency-Key': key },
    data: body,
  });

  expect(replay.ok()).toBeTruthy();
  const payment = await replay.json();
  expect(await ledgerEntryCount(request, { accountId: body.accountId })).toBe(1);
  expect(payment.id).toBe(await paymentIdForKey(request, key));
});

Step 2 carries the weight. Without it, an abort that fires before the server commits turns the whole test into an ordinary first request that happens to be preceded by a dropped connection, and it will pass on a service with no idempotency handling at all. Poll a server-side view until the commit is visible, and fail the test if it never appears rather than proceeding.

Aborting the client is the cheapest technique and not the only one. A proxy that accepts the request, forwards it, and discards the response reproduces a load balancer dropping a connection; a fault-injection hook in the service can commit and then throw before serialising. If your platform supports one of those, prefer it for the scenario where the server commits and then fails to respond, which the client-side abort cannot distinguish.

Four replays that must produce four different answers

The key reused four ways is four separate contracts. A suite that only covers the first has one test wearing four names.

test('the same key with a different payload is refused, not silently ignored', async ({ request }) => {
  const key = randomUUID();
  const first = await request.post('/v1/payments', {
    headers: { 'Idempotency-Key': key },
    data: { accountId: 'acct_7741', amountMinor: 249900, currency: 'INR' },
  });
  expect(first.status()).toBe(201);

  const mismatched = await request.post('/v1/payments', {
    headers: { 'Idempotency-Key': key },
    data: { accountId: 'acct_7741', amountMinor: 999900, currency: 'INR' },
  });

  // The dangerous wrong behaviour is 201 with the ORIGINAL amount: the caller
  // believes 9,999 was charged and 2,499 was. Assert the refusal explicitly.
  expect(mismatched.status()).toBe(422);
  expect(await ledgerTotal(request, 'acct_7741')).toBe(249900);
});

The draft assigns each situation its own status code, and the three are worth writing into the test names because they fail for unrelated reasons: 400 when a documented idempotent operation receives no key at all, 422 when a key is reused with a different payload, and 409 when a retry arrives while the original is still being processed. It also notes that a client must correct the request before retrying in every case except 409, where the right move is to wait and repeat unchanged.

The concurrent case is the one that timeout race needs. Fire both requests without awaiting the first, and assert on the pair rather than on either response:

test('two in-flight requests on one key produce one charge', async ({ request }) => {
  const key = randomUUID();
  const send = () =>
    request.post('/v1/payments', {
      headers: { 'Idempotency-Key': key },
      data: { accountId: 'acct_9930', amountMinor: 120000, currency: 'INR' },
    });

  const [a, b] = await Promise.all([send(), send()]);
  const statuses = [a.status(), b.status()].sort();

  // Either the loser is rejected as a conflict, or it blocks and returns the
  // same result. Which one is a documented choice; both are single-charge.
  expect([[201, 409], [201, 201]]).toContainEqual(statuses);
  if (statuses[1] === 201) {
    expect((await a.json()).id).toBe((await b.json()).id);
  }
  expect(await ledgerTotal(request, 'acct_9930')).toBe(120000);
});

Two requests is the minimum, not the target. Raise it to eight or sixteen once the pair passes, because a lock that serialises correctly for two callers regularly has a window that only opens under more pressure. Keep the ledger assertion identical at every concurrency level: exactly one charge, whatever the statuses were.

Count the effect that was intended, not the rows

RFC 9110 is careful about which effects idempotency covers. The property "only applies to what has been requested by the user", and a server remains "free to log each request separately, retain a revision control history, or implement other non-idempotent side effects for each idempotent request". A test asserting that a replayed request produced exactly one audit row is asserting something the specification explicitly permits the server not to do — and it will fail on a correct implementation.

So pick the counter that represents the operation the caller asked for, and read it from the system of record:

Returning the same identifier is necessary and not sufficient. A service can cache the first response body and replay it faithfully while a second worker completes a second charge, and the identifier assertion will pass through the whole incident. The count is what fails.

When a conditional request is the better mechanism

If the operation modifies a resource that already exists and carries an entity tag, HTTP has a standardised answer to the same problem and you do not need a key at all. RFC 9110 describes If-Match as most often used with state-changing methods to prevent the lost-update problem, and it covers the dropped-response case directly: when a conditional state-changing request "appears to have already been applied to the selected representation", the origin server MAY respond with a 2xx status rather than 412, precisely because the earlier response may have been lost.

The distinction that decides which mechanism you are testing is whether the request creates something new. A capture against an existing order has a resource and an ETag, so the retry can be made safe by the precondition, and the test asserts that a second If-Match request with the original tag returns success or 412 without applying the change twice. A create has no prior representation to match against, which is the gap the key fills. Testing a create with preconditions, or a conditional update with a key layered on top, gives you two mechanisms whose failure modes overlap and a suite that cannot say which one held.

Three ways an idempotency suite passes while production double-charges

Every replay waits for the first response

This is the defect behind the race in the opening, and it is structural rather than accidental: await makes the sequential version the natural one to write. The diagnosis is quick — if no test in the file calls Promise.all or abandons a request, the concurrent path has zero coverage regardless of how many cases there are. The fix is one test, not a rewrite.

The suite passes locally and fails in a parallel run

Two workers generated the same fixture key, or one worker's leftover key from a previous run is still inside the retention window. Generate keys with randomUUID() per test rather than from a seeded pool or a test title, and never reuse a key across a retried Playwright test — a retry with the same key is testing replay, not the scenario the test claims. If keys must come from a pool, take the worker index into account and record which key each attempt used in the failure output.

The tests are green because the middleware is not installed

Idempotency is often a framework filter applied to a route list, and a route added later is simply not on it. A suite full of replay tests against one covered endpoint proves nothing about the endpoint added last month. Add the missing-key case from the draft — a documented idempotent operation with no Idempotency-Key header should answer 400 — and run it against every operation the policy claims to cover. It is one cheap request per route and it detects the unwired endpoint immediately.

Questions that surface once the race is in CI

How do you test a 24-hour retention window in a pipeline?

You do not wait it out, and you should not pretend the test covers it. What is testable is the boundary either side of the window given a way to move the clock: a test-support endpoint that expires a key on demand, or an injectable time source. Then two assertions matter — a replay just inside the window returns the stored result, and the same key after expiry starts a genuinely new operation rather than erroring. If neither hook exists, record retention as untested rather than writing a test that sleeps and asserts nothing.

Does adding a request ID to the body break idempotency?

It depends entirely on how the fingerprint is computed, which is why that was one of the five facts. A checksum over the entire payload treats a new trace ID as a different request and answers 422. A field-value match over selected elements ignores it. Both are described in the draft as acceptable approaches, so neither is a defect — but a client that adds a per-attempt identifier to the body while the server fingerprints the whole payload can never successfully retry, and that combination is worth one explicit test.

Can a model write these scenarios?

It writes the concurrency and mismatch cases well, because they are structural, and it is unreliable on the two things that make them evidence. It does not know your retention policy, your fingerprint fields, or which ledger is the system of record, and it will confidently assert a status code the draft suggests rather than the one your service documents. Use it to expand the scenario list, then replace every expected status with one traced to your API's published policy, and every count assertion with a read against the store that actually holds the money.

Would switching the endpoint to PUT solve this?

Only if the client can choose the resource identifier, and then it solves it properly. PUT is idempotent under RFC 9110, so a client that generates the payment ID and issues PUT /v1/payments/{clientGeneratedId} gets retry safety from the method itself, with no key store and no retention window to reason about. The cost is moving identifier generation to the client and accepting whatever collision handling that implies. If the server must mint the identifier, POST with a key is the design you have, and the tests above are how you hold it.

Primary references

• RFC 9110 §9.2.2, Idempotent Methods: the definition in terms of intended effect, the list of idempotent methods, the allowance for per-request logging and other non-idempotent side effects, and the rules that a client SHOULD NOT auto-retry a non-idempotent method and a proxy MUST NOT
• RFC 9110 §13.1.1, If-Match: preconditions against the lost-update problem, and the permission to answer 2xx when a state-changing request appears already applied because the earlier response was lost
• draft-ietf-httpapi-idempotency-key-header: revision 07 of October 2025, expired 18 April 2026 and not published as an RFC — the source for the Structured Header String syntax (§2.1), key uniqueness (§2.2), published expiry policy (§2.3), fingerprint approaches (§2.4), the retry-versus-concurrent enforcement split (§2.6), and the 400, 422 and 409 error scenarios (§2.7)
• RFC 9651, Structured Field Values for HTTP: the current definition of Item and String, obsoleting the RFC 8941 the draft still cites
• RFC 9110 §15.5.21 and §15.5.10: the meanings of 422 and 409 that the draft's error scenarios rely on
• Playwright — APIRequestContext: the signal option added in v1.62 and used here to abandon a request in flight, and the timeout option whose default is 30000 ms
• Playwright — assertions: expect.poll, used to wait for the server-side commit that makes the first attempt genuinely ambiguous

Apply this now

Open your API's idempotency documentation and try to answer the five facts from it alone. Whichever ones you cannot answer are the tests you cannot write yet, and getting them written down is the higher-value task today.

Then add one test to the operation that moves money or stock: abort the first attempt in flight, poll a server-side view until the commit is visible, replay the key, and assert the ledger total rather than the response identifier. Follow it with the concurrent pair. Afterwards you should be able to point at the key it used, the server-side confirmation that the first attempt committed, both response statuses, and the counter's value on either side of the replay. One caveat on that counter — if it lives in the same service that stores the idempotency key, find one further out before you trust it.