Design Negative API Tests from the Contract

expect(response.status()).toBeGreaterThanOrEqual(400) is the most common assertion in negative API testing and one of the least informative. It passes when the service correctly rejects a malformed order. It also passes when the service crashes, when a proxy returns an HTML error page, and when the route has been deleted.

Replace a validation layer underneath a suite built that way and it stays green through all of it: unknown enum values answering 500 where they used to answer 422, a rejected order that still decremented warehouse stock, another tenant's invoice returning 403 where it previously returned 404 and so confirming which invoice IDs exist. Those are three of the failures a negative test exists to catch, and a 4xx-or-worse assertion is satisfied by every one of them.

One mutation per rule, and put it on the boundary

A case must change exactly one thing relative to a known-valid request, and the change must sit at the edge of the rule it targets rather than somewhere comfortably beyond it. The second half is the part usually skipped. Consider an order line whose quantity must be at least one:

# The rule under test
quantity:
  type: integer
  minimum: 1

A negative case sending quantity: -1 feels thorough and proves very little. JSON Schema 2020-12 defines minimum as an inclusive lower limit and exclusiveMinimum as a strict one, so minimum: 1 and exclusiveMinimum: 0 disagree about exactly one value: zero. A request carrying -1 is rejected under either spelling, so it cannot detect the day someone edits the schema from one to the other. The case that discriminates is quantity: 0, paired with a positive case at quantity: 1.

import { test, expect } from '@playwright/test';
import { validOrder } from './fixtures/valid-order';

// Proves nothing about where the boundary is.
test('rejects a negative quantity', async ({ request }) => {
  const response = await request.post('/v1/orders', {
    data: { ...validOrder, quantity: -1 },
  });
  expect(response.status()).toBe(422);
});

// Pins the boundary: 0 must be rejected and 1 must be accepted.
test('quantity boundary sits between 0 and 1', async ({ request }) => {
  const rejected = await request.post('/v1/orders', {
    data: { ...validOrder, quantity: 0 },
  });
  expect(rejected.status()).toBe(422);

  const accepted = await request.post('/v1/orders', {
    data: { ...validOrder, quantity: 1 },
  });
  expect(accepted.status()).toBe(201);
});

Type rules carry a sharper version of the same trap. JSON Schema's "integer" matches any number with a zero fractional part, so a case built around quantity: 2.0 is accepted, correctly, while a reviewer reading the test name believes non-integer quantities are rejected. The value that exercises the rule is 2.5.

required has a comparable edge: it is satisfied when every listed name appears as a property, and says nothing about the value. So { "quantity": null } satisfies required: ["quantity"] and is rejected by type instead. Sending null to test a required field writes a type case and labels it a presence case. Omit the key.

Read the mutations off the description, defaults included

The enumeration step is mechanical if you take it from the resolved OpenAPI description rather than from the code. Walk each operation and write down every assertion keyword that appears: required, type, enum, minimum and its exclusive twin, minLength, maxLength, pattern, multipleOf, additionalProperties. Each of those is one row. Two OpenAPI defaults add rows nobody writes by hand, because the description never mentions them.

A Request Body Object's required field defaults to false. An operation whose requestBody declares a schema with five required properties but omits required: true has declared that sending no body at all is acceptable — invisible unless a case sends an empty request. A Parameter Object's required defaults to false everywhere except in: path, where the field is mandatory and its value must be true. A filter parameter you assumed was compulsory is optional until somebody writes it down.

# Two rows the keyword walk would miss.
paths:
  /v1/orders:
    post:
      requestBody:
        # No `required: true`, so an empty request is declared valid.
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateOrder'
  /v1/orders/{orderId}:
    get:
      parameters:
        - name: orderId
          in: path
          required: true        # mandatory here, and must be true
          schema: { type: string, format: uuid }
        - name: include
          in: query
          # `required` omitted, so this defaults to optional
          schema: { type: string, enum: [lines, payments] }

format is the one keyword that should not become a rejection case without checking your stack first. Under the Format-Annotation vocabulary that OpenAPI 3.1 uses by default, format is collected as an annotation; implementations may treat it as an assertion, but that behaviour must be disabled by default. A case asserting that format: uuid rejects "not-a-uuid" is testing your validator's configuration, not the contract. Either enable assertion behaviour deliberately and record that you did, or express the rule with pattern so it is enforced by the structural vocabulary.

Four rule families need four different oracles

Rules that look alike in the description have genuinely different correct answers, and the work is choosing the right oracle for each family.

Structural rules — types, bounds, enums, unknown properties — are the only family where a validation error pointer is the oracle. The service knows which member of the instance failed, and the test should assert that location.

References to entities that do not exist are where most matrices adopt the wrong status. If a request to POST /v1/orders names a product SKU that was never created, the target resource of that request is /v1/orders, and it exists. RFC 9110 reserves 404 for the case where the origin server "did not find a current representation for the target resource or is not willing to disclose that one exists", and defines 422 for a request whose content type is understood and whose syntax is correct "but it was unable to process the contained instructions". A dangling reference inside a body is the second case; keep 404 for GET /v1/products/{id} on an ID that does not exist. The HTTP Status Code Reference settles the protocol meaning faster than arguing about it after the test fails.

Authorization rules need an oracle that compares two responses rather than inspecting one. RFC 9110 permits an origin server that wishes to hide the existence of a forbidden resource to answer 404 instead of 403, so the assertion is not "this returns 403" but "this is indistinguishable from the response for an ID that does not exist":

test('another tenant’s invoice is indistinguishable from a missing one', async ({ request }) => {
  // Same identity, two IDs: one real and owned by someone else, one fabricated.
  const foreign = await request.get(`/v1/invoices/${otherTenantInvoiceId}`);
  const absent = await request.get(`/v1/invoices/${randomUuid()}`);

  expect(foreign.status()).toBe(absent.status());
  expect(await foreign.text()).toBe(await absent.text());
  // A body that names the resource, the owner, or the reason re-opens the leak
  // even when both statuses match.
});

One non-disclosure case per resource type is enough here; the full subject-object-action grid across identities, ownership and workflow state is a later part of this series.

State rules cannot be judged from a single request at all. A refund on an order that has not shipped is invalid now and valid later, so the oracle has two halves: the request is refused while the precondition is unmet, and the same request succeeds once the state changes. RFC 9110 describes 409 as a conflict "with the current state of the target resource" that "the user might be able to resolve" and resubmit, and the second half of the test is what proves the code was honest.

test('refund is refused until the order ships, then allowed', async ({ request }) => {
  const { id } = await createOrder(request, { state: 'PLACED' });

  const early = await request.post(`/v1/orders/${id}/refunds`, { data: { amount: 1200 } });
  expect(early.status()).toBe(409);

  await advanceOrderTo(request, id, 'SHIPPED');

  const allowed = await request.post(`/v1/orders/${id}/refunds`, { data: { amount: 1200 } });
  expect(allowed.status()).toBe(201);
});

Assert the failure shape without asserting the sentence

Two specifications agree on this and are routinely ignored together. JSON Schema states outright that for errors, "the specific wording for the message is not defined by this specification" — implementations supply it, and they change it between releases. RFC 9457 says consumers "SHOULD NOT parse the detail member for information" and that extension members are the less error-prone route. The durable oracle is the structured location, never the prose.

RFC 9457's own validation-error example carries an errors extension whose members hold a detail and a pointer locating the problem in the request content with a JSON Pointer. That pointer is what a test should assert.

import { expect, type APIResponse } from '@playwright/test';

/**
 * Asserts the failure contract: the declared media type, agreement between the
 * HTTP status and the advisory `status` member, the problem type, and the exact
 * JSON Pointer of the violation. Never the human-readable strings.
 */
export async function expectProblem(
  response: APIResponse,
  expected: { status: number; type: string; pointer?: string },
) {
  expect(response.status()).toBe(expected.status);
  expect(response.headers()['content-type']).toContain('application/problem+json');

  const problem = await response.json();

  // RFC 9457 3.1.2: `status` is advisory, but a generator MUST use the same code
  // in the response. A mismatch means an intermediary rewrote one of them.
  if ('status' in problem) expect(problem.status).toBe(expected.status);

  // RFC 9457 3.1.1: an absent `type` is assumed to be "about:blank", which
  // identifies nothing. Requiring an explicit type is what makes the case stable.
  expect(problem.type ?? 'about:blank').toBe(expected.type);

  if (expected.pointer) {
    const pointers = (problem.errors ?? []).map((e: { pointer?: string }) => e.pointer);
    expect(pointers).toContain(expected.pointer);
  }
}

The status check looks redundant and is not. RFC 9457 requires a generator to use the same code in the actual HTTP response, and notes in its security considerations that generic HTTP software — proxies, load balancers, virus scanners — will not respect the code carried in the body. A disagreement between the two is evidence that something in the path rewrote the response, which is a finding rather than a flake.

If your service returns a bare validator dump instead, its field names are not portable either. JSON Schema's own output units use instanceLocation and keywordLocation, the specification requires only one of the flag, basic or detailed structures, and popular validators name the same idea differently. Pick the shape your gateway emits, write it into the helper once, and assert against the helper.

Prove the rejection left nothing behind

This is the assertion behind the second failure in the opening. A 422 tells you the request was refused. It does not tell you whether the handler had already reserved stock, written an audit row, incremented a rate-limit counter, or sent a confirmation email before the validation error surfaced. Each rejected case needs a probe chosen for what that operation would have touched:

test('a rejected order reserves no stock and creates no order', async ({ request }) => {
  const sku = await seedProduct(request, { stock: 5 });
  const before = await stockLevel(request, sku);
  const ordersBefore = await orderCount(request, { sku });

  const response = await request.post('/v1/orders', {
    data: { ...validOrder, sku, quantity: 0 },
  });
  await expectProblem(response, {
    status: 422,
    type: 'https://errors.example.com/validation',
    pointer: '/quantity',
  });

  expect(await stockLevel(request, sku)).toBe(before);
  expect(await orderCount(request, { sku })).toBe(ordersBefore);
});

The probes that catch real defects read a durable counter: rows in a table exposed through a support endpoint, a ledger total, the length of a list response. A probe that re-reads the resource you just failed to create catches only the crudest partial write, because the object it looks for was never given an identifier.

Multi-step operations are where partial writes live. If creating an order writes the order, reserves stock, then charges a card, a rule enforced at the third step has two earlier writes behind it. Cases whose rule is enforced late need a probe on every earlier step, and the surrounding transaction does not cover work done by another service.

The matrix is a reviewable file, and it costs you something

Keeping the enumeration in a data file rather than in 40 hand-written test bodies is what makes the coverage argument reviewable: someone can diff the matrix against the schema and see which keywords have no row.

// contracts/create-order.negatives.ts
export const negatives = [
  { rule: 'quantity/minimum', mutate: (o) => ({ ...o, quantity: 0 }),
    status: 422, pointer: '/quantity', probe: 'stock' },
  { rule: 'quantity/type', mutate: (o) => ({ ...o, quantity: 2.5 }),
    status: 422, pointer: '/quantity', probe: 'stock' },
  { rule: 'sku/required', mutate: ({ sku, ...rest }) => rest,
    status: 422, pointer: '/sku', probe: 'none' },
  { rule: 'currency/enum', mutate: (o) => ({ ...o, currency: 'XTS' }),
    status: 422, pointer: '/currency', probe: 'none' },
  { rule: 'body/required', mutate: () => undefined,
    status: 400, pointer: undefined, probe: 'none' },
  { rule: 'sku/reference', mutate: (o) => ({ ...o, sku: 'SKU-NEVER-CREATED' }),
    status: 422, pointer: '/sku', probe: 'orders' },
] as const;

The cost is real. Every row couples to your error contract, so a deliberate move of validation from 422 to 400 breaks every row at once — correct behaviour that still arrives as a wall of red. Boundary cases also break when the boundary legitimately moves, and that failure looks identical to a regression until someone reads the schema diff. Comparing the two schema versions in the JSON Diff tool tells you within a minute whether the contract moved or the service did.

The row that does not pay for itself is the fuzzed one. Broad random payloads find anomalies, but with no rule name attached to an input there is no oracle: a 500 from a mutated byte says something is wrong, not which promise the service failed to keep. Keep fuzzing if you have it, and keep it out of the matrix.

When a negative case is quietly lying

Every case passes, including against a service that is not running

A family assertion is satisfied by connection failures, gateway errors and HTML error pages. Add one canary: a valid request that must return its documented success code. If the canary fails, the negative results in that run mean nothing. This is also why the exact status matters — a 503 from a crashed pod does not resemble a 422.

The case passes locally and fails behind the gateway

Two validators are enforcing the contract and they disagree about which rejects first. A gateway validating against the published description rejects on required and type before the application sees the request, and its problem body will not carry your application's type URI. Decide which layer owns each rule family, then run the matrix through the same ingress path CI uses. A matrix that only runs against the application process is measuring a component, not the API.

The status is right and the pointer is empty

The service rejected the request without saying where. A caller cannot turn that into a field-level message, and it usually means the rejection came from a hand-written guard clause rather than from schema validation — in which case the other keywords on that property are unenforced. The JSON Schema Generator turns a representative payload into a schema you can compare against what the service enforces, though only the description settles what it should enforce.

Questions that arrive with the first real matrix

Should validation failures be 400 or 422?

Pick one per API and write it into the matrix rather than deciding case by case. RFC 9110 defines 400 for a request the server will not process "due to something that is perceived to be a client error", including malformed syntax, and 422 for content whose media type is understood and whose syntax is correct but whose instructions cannot be processed. Unparseable JSON is 400 under any reading. A schema violation in well-formed JSON fits 422 more precisely, though 400 stays defensible and many gateways emit it. A mix is what breaks the matrix, because then no test can assert an exact code.

How many negative cases does one operation need?

One per assertion keyword in its schema, one per referenced entity, one per state precondition, and one non-disclosure case per resource type it exposes. A six-property create operation lands between twelve and twenty. Three means the schema has rules nobody enforces; ninety means you are testing combinations, and a combination cannot name the rule that failed.

What should the service return when a request breaks three rules at once?

A contract decision to document, not a case to add. RFC 9457 recommends representing the most relevant or urgent problem when problems do not share a type, and its validation example shows the alternative: one problem type with an errors extension listing every occurrence. Assert whichever your API promises, and keep the rows single-mutation regardless — a row breaking three rules cannot say which one the service stopped enforcing.

Can a model generate the matrix from the schema?

It is good at the enumeration and unreliable at the oracle. A resolved schema produces a usable first list of keywords and boundary values, often catching a pattern or maxLength you skipped. What it must not decide is the expected status for a family, the pointer your service emits, or which probe proves a side effect is absent — those come from the description, from an observed response, and from knowing what the handler writes. Take the generated list as rows and supply the three columns that make a row an oracle.

Is an unknown property in the request a negative case?

Only if the schema closes the object. With additionalProperties omitted the contract accepts unknown members, so a case asserting rejection tests a rule the API never declared. The better question is whether the unknown property is silently bound: if sending {"role": "admin"} to a profile update changes anything, that is a property-level authorization defect, and the probe belongs on the field it wrote.

Primary references

• RFC 9110, HTTP Semantics — Client Error 4xx: the definitions this article relies on for 400 (§15.5.1), 403 including the permission to answer 404 to hide existence (§15.5.4), 404 (§15.5.5), 409 (§15.5.10), and 422, which RFC 9110 moved out of WebDAV into core HTTP (§15.5.21)
• RFC 9457, Problem Details for HTTP APIs: the application/problem+json media type, the advisory status member and the requirement that it match the response (§3.1.2), the absent-type default of about:blank (§3.1.1), the instruction not to parse detail (§3.1.4), extension members (§3.2), and the errors/pointer validation example
• JSON Schema 2020-12 Validation: integer matching any number with a zero fractional part (§6.1.1), inclusive minimum against exclusive exclusiveMinimum (§6.2.4–6.2.5), required checking property presence only (§6.5.3), and the Format-Annotation vocabulary requiring assertion behaviour to be off by default (§7.2.1)
• JSON Schema 2020-12 Core: output units and their instanceLocation and keywordLocation keys (§12.3), the four output structures and the fact that only one of flag, basic or detailed is required (§12.2), and the statement that error wording is not specified (§12.3.4)
• OpenAPI Specification 3.1.1: Request Body Object required defaulting to false (§4.8.13.1) and Parameter Object required defaulting to false except in path (§4.8.12.2.1)
• OWASP API Security Top 10 2023 — API3:2023: why an accepted unknown property is an authorization question, and the merge of the former excessive-data-exposure and mass-assignment categories
• Playwright — API testing: the request fixture and APIResponse members used in the examples, checked against Playwright 1.62

Apply this now

Take the operation in your API with the most required properties. Open its resolved schema, list every assertion keyword on it, and count how many of those keywords have a test that fails when the keyword is deleted. That count, not the number of negative tests, is your coverage.

Then fix one row end to end: move its mutation onto the boundary, replace the family assertion with the exact status, assert the JSON Pointer instead of the message, and probe the durable counter the handler would have touched. Add the canary in the same commit. What should exist on the other side of that work is a keyword list with a covered-or-not column, a recorded pointer and probe for each case, and a settled answer to which status your API uses for malformed syntax and which it uses for schema violations. If that last one is still open, settle it before writing the second row — every row you add before then will need editing afterwards.