API Contract Testing with Playwright and JSON Schema

Take a payments service that returns an order summary. In a minor release the field amountDue is renamed to amount_due. The consumer's contract suite runs against the deployed service on every pipeline and reports twelve green checks. The web front end starts rendering an empty total two days later, and the schema that was supposed to prevent exactly this had the field written into it the whole time.

The schema listed amountDue under properties and left it out of required, because in a draft order it can legitimately be absent. So the rename produced two effects that cancelled out. The old key was gone, and nothing required it. The new key was unrecognised, and unrecognised keys are permitted. Both halves of the failure were invisible to the validator, and the sum of two invisible failures is a pass.

A request suite that never starts a browser

Playwright's test runner ships a request fixture that speaks HTTP directly from Node. The documentation is explicit that it "respects configuration options like baseURL or extraHTTPHeaders", and that behind the scenes it calls apiRequest.newContext() for you. Nothing launches, so a contract project runs in the time it takes to do the round trips.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'contract',
      testDir: './tests/contract',
      use: {
        baseURL: process.env.ORDERS_API ?? 'http://localhost:8080',
        extraHTTPHeaders: {
          Accept: 'application/json',
          Authorization: `Bearer ${process.env.ORDERS_TOKEN}`,
        },
      },
    },
    // ...browser projects live alongside and are unaffected.
  ],
});

Per-file overrides use test.use(), which the options reference documents as the way to override some options for a file. That matters when one endpoint sits behind a different gateway or needs a different media type:

// tests/contract/orders.spec.ts
import { test, expect } from '@playwright/test';

test.use({ extraHTTPHeaders: { Accept: 'application/vnd.orders.v2+json' } });

test('GET /orders/{id} answers with a summary', async ({ request }) => {
  const response = await request.get('/orders/8842');

  // toBeOK() asserts the status is inside 200..299 — nothing about the body.
  await expect(response).toBeOK();
  expect(response.headers()['content-type']).toContain('application/json');

  const body = await response.json();
  expect(body.id).toBe('8842');

  // Response bodies stay in memory until the context closes. Long files that
  // fetch large payloads should hand them back explicitly.
  await response.dispose();
});

expect(response).toBeOK() and response.json() are the whole of what Playwright offers about payload correctness: a status range and a parsed object. Every claim about the shape of that object has to come from somewhere else.

The schema that cannot fail

Here is the check that let the rename through, written out in full. Both payloads below satisfy it.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "id":        { "type": "string" },
    "status":    { "type": "string", "enum": ["draft", "open", "settled"] },
    "amountDue": { "type": "number" },
    "currency":  { "type": "string" }
  },
  "required": ["id", "status"]
}

// Valid — the shape everyone believes is being enforced.
{ "id": "8842", "status": "open", "amountDue": 4150, "currency": "INR" }

// Also valid. amountDue is gone; amount_due is an unrecognised property,
// and unrecognised properties are allowed by default.
{ "id": "8842", "status": "open", "amount_due": 4150, "currency": "INR" }

The specification's own words for this are that "by default any additional properties are allowed", and separately that "by default, the properties defined by the properties keyword are not required". Listing a property is a conditional statement: if this key is present, it must look like this. It is never an instruction that the key must exist. Two keywords, and neither of them is doing what the author assumed.

The closed version differs by two lines and behaves completely differently:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "id":        { "type": "string" },
    "status":    { "type": "string", "enum": ["draft", "open", "settled"] },
    "amountDue": { "type": ["number", "null"] },
    "currency":  { "type": "string", "minLength": 3, "maxLength": 3 }
  },
  "required": ["id", "status", "amountDue", "currency"],
  "additionalProperties": false
}

Now the rename fails twice over: required reports a missing amountDue, and additionalProperties reports amount_due as a property that does not belong. Note also what happened to nullability. A draft order has no amount yet, but the field is still sent, so the type becomes ["number", "null"] and the name goes into required. The reference documentation states the rule plainly: in JSON a property whose value is null is not equivalent to the property not being present. Modelling "sometimes empty" as "sometimes missing" is what pushed the field out of required in the first place, and that single modelling choice is what disarmed the check.

Configuring Ajv so failures are loud

Ajv compiles a schema into a JavaScript validation function, which is why the compile step is comparatively slow and the validation step is not. Its own guidance is to compile once and reuse the returned function — module scope is the natural home for that.

// tests/contract/schema.ts
import Ajv2020 from 'ajv/dist/2020';
import addFormats from 'ajv-formats';
import type { ErrorObject, ValidateFunction } from 'ajv';

// strictSchema defaults to true, and leaving it alone is the point: it turns a
// typo such as "requried" from a silently ignored keyword into a compile throw.
const ajv = new Ajv2020({
  allErrors: true, // default is false: reporting stops at the first error
  verbose: true,   // attaches the offending data to each error object
});

// Ajv 7 and later ship no formats at all. Without this line, "format": "uuid"
// throws during compilation rather than being quietly skipped.
addFormats(ajv);

export const compile = (schema: object): ValidateFunction => ajv.compile(schema);

export const explain = (errors: ErrorObject[] | null | undefined): string =>
  (errors ?? [])
    .map((e) => {
      const where = e.instancePath || '(root)';
      const extra = JSON.stringify(e.params);
      return `${where} ${e.keyword}: ${e.message} ${extra}`;
    })
    .join('\n');

Three details in there change what your failures look like. allErrors defaults to false, so an unconfigured validator reports one problem per run and hides the other four. verbose adds the failing data, schema and parentSchema to each error object. And the params object is keyword-specific in a way that is exactly what you want to print: required errors carry missingProperty, and additionalProperties errors carry additionalProperty — the name of the key nobody expected.

One trap deserves its own sentence, because Ajv's getting-started guide flags it and it survives code review easily: every call to a validation function overwrites the errors property. Capture it immediately, or the assertion you eventually write will report the wrong response's problems.

// tests/contract/orders.spec.ts
import { test, expect } from '@playwright/test';
import { compile, explain } from './schema';
import orderSummary from './schemas/order-summary.json';

const validateOrder = compile(orderSummary);

test('the order summary matches the published contract', async ({ request }) => {
  const response = await request.get('/orders/8842');
  await expect(response).toBeOK();

  const body = await response.json();
  const valid = validateOrder(body);
  // Copy the reference before anything else can run the validator again.
  const errors = validateOrder.errors ? [...validateOrder.errors] : [];

  expect(valid, `contract violated:\n${explain(errors)}`).toBe(true);
});

Taking the schema from the OpenAPI document instead

Hand-maintained schemas drift, and a drifted schema is a check that agrees with whatever it was last edited to agree with. If the provider publishes an OpenAPI 3.1 document, you can validate against that document directly, because the specification defines its Schema Object as "a superset of JSON Schema Specification Draft 2020-12". The extras it adds are the OAS base vocabulary — discriminator, xml, externalDocs, example and deprecated — plus permission for arbitrary further keywords.

That permission collides with Ajv's strict mode, which fails compilation on unknown keywords by design, since the alternative is that a mistyped keyword is ignored and the schema quietly weakens. Declare the OAS vocabulary rather than switching strictness off:

// tests/contract/from-openapi.ts
import { readFileSync } from 'node:fs';
import YAML from 'yaml';
import Ajv2020 from 'ajv/dist/2020';
import addFormats from 'ajv-formats';

const ajv = new Ajv2020({ allErrors: true });
addFormats(ajv); // supplies int32, int64, binary, byte — the OAS format values

// Declared as known-and-ignorable. If you want Ajv to actually enforce tagged
// unions, drop 'discriminator' from this list and pass discriminator: true,
// which defaults to false.
ajv.addVocabulary(['discriminator', 'xml', 'externalDocs', 'example', 'deprecated']);

const doc = YAML.parse(readFileSync('openapi.yaml', 'utf8'));

// Component refs are JSON Pointers into the OpenAPI document. Re-root them so
// the component map can be registered as one ordinary schema resource.
const components = JSON.parse(
  JSON.stringify({ $defs: doc.components.schemas }).replaceAll(
    '#/components/schemas/',
    '#/$defs/',
  ),
);

export const schemaFor = (name: string) =>
  ajv.compile({ ...components, $ref: `#/$defs/${name}` });

OpenAPI 3.0 needs a conversion step first: that version defines its Schema Object as "an extended subset of JSON Schema Specification Draft Wright-00", with nullable: true standing in for a null type. Ajv understands nullable and, behind an option, discriminator as OpenAPI extensions, but you cannot mix draft 2020-12 and earlier drafts inside one Ajv instance, so a 3.0 document wants its own instance built from the default draft-07 export.

There is a limit to this technique that is worth stating rather than discovering. If the provider generates its OpenAPI document from the same annotations that produce the response, a rename updates the document and the test, and the pair stay agreeable while your client breaks. Pin the version of the spec your consumer was written against, and treat a diff in it as a review item, not as an automatic upgrade.

Composition quietly reopens a closed schema

Provider specs love allOf. A base Order extended by SettledOrder is the idiomatic OpenAPI way to express that, and it breaks additionalProperties outright. The reference documentation states the constraint precisely: additionalProperties "only recognizes properties declared in the same subschema as itself". Everything contributed by a branch of allOf is, from the outer schema's point of view, additional.

// Wrong: nothing can satisfy this. "settledAt" is required by the outer
// schema and rejected by the inner one, which never heard of it.
{
  "allOf": [
    {
      "type": "object",
      "properties": { "id": { "type": "string" }, "status": { "type": "string" } },
      "required": ["id", "status"],
      "additionalProperties": false
    }
  ],
  "properties": { "settledAt": { "type": "string", "format": "date-time" } },
  "required": ["settledAt"]
}

// Right: unevaluatedProperties collects what the subschemas successfully
// validated and rejects only what nothing accounted for.
{
  "allOf": [
    {
      "type": "object",
      "properties": { "id": { "type": "string" }, "status": { "type": "string" } },
      "required": ["id", "status"]
    }
  ],
  "properties": { "settledAt": { "type": "string", "format": "date-time" } },
  "required": ["settledAt"],
  "unevaluatedProperties": false
}

The keyword arrived in draft 2019-09, which is the practical reason to run Ajv's 2020 class for this work even when your schemas use nothing else from the newer drafts. Teams that hit the broken version usually respond by deleting additionalProperties, and land back at an open schema without noticing they have given up the only check that catches a stray field.

Where Pact answers a different question

Schema validation asks whether one response matches a published description. Consumer-driven contract testing asks whether a provider still satisfies every consumer that depends on it, and Pact's documentation is careful about the boundary. Each interaction records "a minimal expected response — describing the parts of the response the consumer wants the provider to return", and provider verification "passes if each request generates a response that contains at least the data described in the minimal expected response".

Read that literally and the honest comparison falls out. Pact does not fail on an added field either; that is deliberate, and its authoring guidance says to "only make assertions about things that will affect the consumer if they change". So additionalProperties: false in your own test suite remains the only mechanism here that notices a field arriving unannounced — and it is a check on your understanding of the API, not a constraint you are entitled to impose on the provider.

What Pact adds is the direction of authority. The expectations come from real consumer code paths across all consumers, so a provider learns before deploying that a field it considered internal is being read by two teams. Its docs also mark out what it deliberately does not do: "a contract test does not check for side effects", and functional questions such as whether the order was actually persisted belong to the provider's own tests.

Held together, the split is simple. Schema checks give you fast, browser-free verification that a live environment still answers in the shape you were promised, and they run against staging today with no coordination. Pact gives you a broker, a versioned contract, and a can-I-deploy answer, at the cost of both teams adopting it. The first is a test you can add this afternoon; the second is a process change.

Apply this now

Open the schemas your contract suite already uses and grep them for additionalProperties. For each one that does not have it, add "additionalProperties": false, run the suite against staging, and keep the output. That run is the evidence worth capturing: every failure it produces is a field your service is sending today that your contract does not describe, which is the drift you have been carrying unmeasured.

Then work through the failures in two piles. Fields that consumers actually read go into properties and into required, with ["number", "null"]-style types where the value can be empty. Fields nobody reads get left out and stay a deliberate failure until someone decides. Where a schema uses allOf, swap in unevaluatedProperties: false instead, and confirm the swap by adding a junk key to a fixture and watching the test go red.

Frequently asked questions

Should every schema close itself with additionalProperties: false?

In a consumer's test suite, yes — that is where you want to be told about anything unexpected. In a schema you publish as the definition of your own API, no: closing it stops any client from tolerating fields you add later, and Pact's guidance against over-constraining a provider applies for the same reason. The distinction is whether the schema is a detector or a promise.

Do these tests need a browser installed?

No. The request fixture issues HTTP from Node without loading a page, which is why the documented use cases include preparing server-side state before a browser test and checking server-side post-conditions afterwards. Keeping them in a separate project with its own testDir lets the contract suite run on pipelines where browsers are not provisioned.

Why does Ajv throw "unknown format" on a schema that worked before?

From version 7 onwards Ajv ships none of the formats defined by the JSON Schema specification; they live in the ajv-formats package. Unknown formats throw during schema compilation by default, which is a feature — the alternative is "format": "date-tiem" passing everything. Call addFormats(ajv), or register the individual names with addFormat.

Is a field set to null the same as a missing field?

No, and conflating them is the most common way a required list gets hollowed out. A null value is present and has type "null", so it satisfies required and fails {"type": "string"}. Express it as {"type": ["string", "null"]} and keep the name in required, rather than dropping the name to make the failure go away.

Where should the schemas live — with the tests or with the API?

With the consumer, under version control, even when they are generated from the provider's OpenAPI document. A schema fetched fresh from the provider at run time cannot detect drift, because it changes at the same moment the response does. Regenerate deliberately, commit the diff, and let review decide whether the change was intentional.

Primary references

• Playwright — API testing: the request fixture respecting baseURL and extraHTTPHeaders, its equivalence to apiRequest.newContext(), and browser-free usage
• Playwright — TestOptions: baseURL resolution rules, extraHTTPHeaders, and overriding options for a single file with test.use()
• Playwright — APIResponse: json(), headers(), status(), and dispose() releasing a body held in memory
• Playwright — APIResponseAssertions: toBeOK() covering the 200–299 status range and nothing more
• Playwright — Assertions: passing a custom failure message as the second argument to expect
• JSON Schema — object: additional properties allowed by default, additionalProperties: false, required being independent of properties, null versus absence, the same-subschema restriction under allOf, and unevaluatedProperties
• Ajv — Strict mode: unknown keywords and unknown formats failing schema compilation, addVocabulary, and strictRequired
• Ajv — Options: the option defaults table, including allErrors: false, verbose: false, strictSchema: true and discriminator: false
• Ajv — API reference: compile, the ErrorObject fields, and the keyword-specific params including missingProperty and additionalProperty
• Ajv — Getting started: compilation cost versus validation cost, and errors being overwritten on every call
• Ajv — JSON Schema: the draft 2020-12 class import, drafts not mixing in one instance, and support for the OpenAPI nullable and discriminator keywords
• ajv-formats: addFormats(ajv) and the format list, including the OpenAPI values int32, int64, byte and binary
• OpenAPI Specification 3.1.0: the Schema Object as a superset of JSON Schema draft 2020-12, and the base vocabulary keywords the dialect adds
• OpenAPI Specification 3.0.4: the Schema Object as an extended subset of Draft Wright-00, and nullable in place of a null type
• Pact — How Pact works: minimal expected responses, and provider verification passing on "at least" the described data
• Pact — Contract tests vs functional tests: contract tests not checking side effects, and the table of which test owns which assertion
• Pact — Writing consumer tests: asserting only on what affects the consumer if it changes