Playwright MCP Is Not Your Regression Test Suite

An agent with Playwright MCP opens a checkout page, types a coupon, presses Apply and confirms the message "Coupon SAVE20 applied". It hands back the code it ran, and that code becomes a test. In the lab behind this article, the test passed 20 times out of 20 against the correct page. It also passed 20 out of 20 against three broken versions: one that dropped a coupon applied before the cart had loaded, one that never subtracted the discount, and one that sent the coupon under the wrong query parameter, so the server rejected it.

The page really did say the coupon was applied, and the session proved only that. A regression suite has to answer whether the total is right whatever order the responses arrive in. This finale of the series places Playwright MCP and Playwright's own test agents on either side of that line. The Appium MCP article covers the mobile side; everything below is browser-specific.

What the model reads: a snapshot with refs

Playwright MCP does not send the model a screenshot to act on. After navigation and most actions it returns an accessibility snapshot: a YAML tree of roles, accessible names and values, with a ref on each node. The model passes a ref back as the target of the next call. This is the snapshot the lab page produced after the coupon was applied:

- main [ref=e2]:
  - heading "Checkout" [level=1] [ref=e3]
  - paragraph [ref=e4]:
    - text: "Order total:"
    - status [ref=e5]: $100.00
  - generic [ref=e6]:
    - generic [ref=e7]:
      - text: Coupon code
      - textbox "Coupon code" [ref=e8]: SAVE20
    - button "Apply coupon" [active] [ref=e9]
  - status [ref=e10]: Coupon SAVE20 applied

The total is an <output> element, which the tree reports as status, so the page has two unnamed elements with that role and a role-only locator would be ambiguous. Refs last only until the page changes; the snapshot documentation says a stale one fails with an instruction to take a new snapshot. That suits an agent and is useless as an identifier in a committed test. Each action also returns the Playwright code it ran, such as await page.getByRole('button', { name: 'Apply coupon' }).click();, and those lines are the raw material for any test the agent writes.

The default server registers 25 tools in 0.0.83, already including browser_evaluate and browser_run_code_unsafe, which run arbitrary script in the page or Playwright code against it. Network mocking, storage, tracing, coordinate clicks, PDF and the verification tools are opt-in through --caps; with every capability on, the lab server listed 72 tools.

The agent's browser is not your CI browser

@playwright/mcp 0.0.83 does not depend on a released Playwright: its package.json pins playwright 1.64.0-alpha-1790635538000, with Chromium revision 1247 (155.0.8059.12), while @playwright/test 1.64.1 expects revision 1248, Chrome for Testing 156.0.8078.4. And the agent drives neither by default. The MCP configuration code assigns the chrome channel when no browser is named, so it launches the machine's installed Google Chrome. In the lab, both the standalone server and the bundled copy reported "Google Chrome 155" from navigator.userAgentData; the test runner's chromium.launch() reported 156.0.8078.4.

Since 1.62, Playwright bundles the server as npx playwright mcp. Run it from the same pinned @playwright/test as your suite with --browser chromium, which maps to Chrome for Testing, and the agent explores on the build a default Chromium project tests on.

A server configured to leave evidence

By default the server keeps a persistent profile in the user's cache directory, keyed by a hash of the client's workspace, so logins survive between sessions. The extension mode goes further and attaches to your own running Chrome. --isolated keeps the profile in memory and discards it when the browser closes, which is the only mode with a starting state you can describe; --storage-state seeds it with a known test user.

{
  "mcpServers": {
    "checkout-explore": {
      "command": "npx",
      "args": [
        "playwright", "mcp",
        "--browser", "chromium",
        "--isolated",
        "--storage-state", "explore/test-user.json",
        "--allowed-origins", "https://staging.example.test",
        "--caps", "devtools,testing",
        "--save-session",
        "--output-dir", "explore/out",
        "--secrets", "explore/.env.explore"
      ]
    }
  }
}

Two of those flags are weaker than their names suggest. --allowed-origins blocks requests from other origins (in the lab, a navigation to http://localhost:4387/ with only the 127.0.0.1 origin allowed failed with net::ERR_BLOCKED_BY_CLIENT), but the CLI help says it is not a security boundary and does not apply to redirects. Treat it as a way to keep the session on topic, and put the real control in the network the browser runs on; the SSRF part of this series covers why redirect-blind allowlists fail. --secrets replaces matching values in tool responses so the model does not read them, and the configuration schema calls it a convenience, not a security feature.

The page is untrusted input too. Text on a page someone else controls can steer the agent, and with browser_run_code_unsafe in the default set the damage need not stay in the page; the prompt-injection part shows how to test for that. Explore on an environment you own, with a test account worth nothing, and never against production with write access. The MCP security guidance for local servers adds the other half: run the server sandboxed, with minimal default access to files and network.

--save-session writes a Markdown log of every tool call and its result, and --caps=devtools adds tools that record a Playwright trace while the agent works. Those two files are the evidence everything below depends on. If they go to a log store, the observability part covers keeping credentials out of them.

The emitted assertion is weaker than the check

With --caps=testing, the agent gets verification tools, and Playwright's MCP documentation says each one records the matching expect(...) line, so that a passing sequence can be assembled into a test. Here is the call from the opening, and what it recorded:

>>> browser_verify_text_visible {"text":"Coupon SAVE20 applied"}
### Result
Done
### Ran Playwright code
```js
await expect(page.getByTestId('coupon-status')).toBeVisible();
```

The text is gone. The tool did search the page for "Coupon SAVE20 applied", but then, in verify.ts at v1.64.1, it normalises the matching locator to the best one it can generate, here the test id of the status paragraph, and records a visibility assertion on that. The recorded line passes for any visible message in that paragraph, including "Coupon not recognised". That is why the session test survived all three broken versions.

// recorded by the session: passes whatever the message says
await expect(page.getByTestId('coupon-status')).toBeVisible();

// what the check meant: the message and the money
await expect(page.getByTestId('coupon-status')).toHaveText('Coupon SAVE20 applied');
await expect(page.getByTestId('order-total')).toHaveText('$100.00');

Reading generated specs for assertions that cannot fail is a habit covered in the review checklist for AI-written Playwright tests. The narrower point: "Done" in the session and the line it hands you are two different checks, so review the line.

What the session log shows, and what it hides

The lab server answers /api/cart after a random delay of up to 400 ms, as a slow cart service would. The saved session log shows the agent arriving while that request was still in flight:

### Tool call: browser_navigate
- Result
  "snapshot": "... - status [ref=e5]: Loading… ..."

### Tool call: browser_click
- Result
  "code": "await page.getByRole('button', { name: 'Apply coupon' }).click();",
  "snapshot": "... - status [ref=e5]: $100.00 ... - status [ref=e10]: Coupon SAVE20 applied"

By the time the click ran, the cart had answered. Each MCP action waits for the page to settle before returning, 500 ms by default under --timeout-settle, and a real model adds thinking time between calls. At that pace the coupon nearly always lands after the cart, and the agent walks past a bug that needs the opposite order.

The test runner works at machine speed. The session's three action lines plus an assertion on the total, run with --repeat-each=20 against the shipped page, failed 17 times and passed 3. The trace of one failure, read with the npx playwright trace commands added for agents in 1.59, shows the order:

$ npx playwright trace requests
     # Start     Method   Status   Name                                            Duration     Size Route
  ──── ───────── ──────── ──────── ───────────────────────────────────────────── ────────── ──────── ──────────
    1. 0:00.291  GET      200      127.0.0.1:4388                                      12ms      628
    2. 0:00.312  GET      200      app.js                                              10ms     1.1K
    3. 0:00.341  GET      200      cart                                               351ms      171
    4. 0:00.457  GET      200      coupon?code=SAVE20                                   2ms      183

The coupon answered at about 0.46 s; the cart was pending until about 0.69 s. The MCP session's own trace, opened the same way, has the cart finishing at 0.32 s and the coupon starting at 0.40 s. Same page, opposite orders. The reviewed root cause, written down where someone can dispute it: the shipped code applies the discount only if the subtotal is already known, and never applies it later. The trace-viewer workflow covers reading a trace to that depth.

That candidate is still not the regression test. Failing 17 times in 20, it would look flaky in CI, earn a retry and pass on it. The regression has to produce the bad ordering every time.

Hold the cart response until the coupon has answered

The contract is that the total is right in either order. Encode both orders, and force the dangerous one with a route that holds the cart request until the coupon response has arrived:

import { test, expect } from '@playwright/test';

test.describe('coupon discount, whatever order the responses arrive in', () => {
  test('a coupon applied before the cart loads still discounts the total', async ({ page }) => {
    let releaseCart: () => void = () => {};
    const cartHeld = new Promise<void>((resolve) => { releaseCart = resolve; });
    await page.route('**/api/cart', async (route) => {
      await cartHeld;
      await route.continue();
    });

    await page.goto('/');
    const couponAnswered = page.waitForResponse((r) => r.url().includes('/api/coupon'));
    await page.getByRole('textbox', { name: 'Coupon code' }).fill('SAVE20');
    await page.getByRole('button', { name: 'Apply coupon' }).click();
    await couponAnswered;
    releaseCart();

    await expect(page.getByTestId('coupon-status')).toHaveText('Coupon SAVE20 applied');
    await expect(page.getByTestId('order-total')).toHaveText('$100.00');
  });

  test('a coupon applied after the cart loads discounts the total', async ({ page }) => {
    await page.goto('/');
    await expect(page.getByTestId('order-total')).toHaveText('$120.00');
    await page.getByRole('textbox', { name: 'Coupon code' }).fill('SAVE20');
    await page.getByRole('button', { name: 'Apply coupon' }).click();
    await expect(page.getByTestId('order-total')).toHaveText('$100.00');
  });
});

route.continue() lets the real server produce the cart; the test controls when it arrives, not what it says. One gap remains: a received coupon response does not prove the page has handled it before the cart is released. If your page shows the coupon result before the total is known, assert that message before releaseCart() to pin the order completely; as written, this test relies on the response. toHaveText retries until the text matches or the five-second default timeout expires, so no sleep is needed, and the second test waits for $120.00 before typing to guarantee the ordinary order.

Against the shipped page the first test fails on every run. The call log shows it is no timing accident: the total went from "Loading…" to the undiscounted amount and never showed the right one.

Error: expect(locator).toHaveText(expected) failed

Locator:  getByTestId('order-total')
Expected: "$100.00"
Received: "$120.00"
Timeout:  5000ms

Call log:
  - Expect "toHaveText" getByTestId('order-total') with timeout 5000ms
  - waiting for getByTestId('order-total')
    4 × locator resolved to <output data-testid="order-total">Loading…</output>
      - unexpected value "Loading…"
    10 × locator resolved to <output data-testid="order-total">$120.00</output>
       - unexpected value "$120.00"

Run every candidate against mutants before merge

A test that has only ever passed has not shown it can detect anything. Before it joins the suite, run it against small, deliberate breakages of the code it protects. These three were written by hand, each a plausible way to bring the bug back or break the feature beside it:

• M1, the shipped bug: the discount is applied only if the subtotal is already known.
• M2: the render ignores the discount.
• M3: the coupon is sent as ?coupon= instead of ?code=, so the server rejects it.

for app in app.js app-race.js app-nodiscount.js app-wrongparam.js; do
  APP_JS=$app PLAYWRIGHT_JSON_OUTPUT_NAME=results-$app.json \
    npx playwright test --repeat-each=20 --reporter=json > /dev/null
done

The webServer block in the Playwright config starts the page with APP_JS taken from the environment, so each loop serves one variant. Failures out of 20 runs per cell:

By column, each mutant must be killed on every run by at least one test, and only the held-cart test kills M1 reliably. The after-cart test surviving M1 is fine; the pair covers the contract. By row, zeros under every mutant mark a test that cannot fail, whatever its name says. Keep the mutants beside the spec, as files or patches, and rerun the loop before merging any change to it.

Playwright's planner, generator and healer

Playwright 1.56 introduced three test agents. npx playwright init-agents --loop=claude (also vscode, codex, copilot and opencode) writes a planner, a generator and a healer as agent definitions, a seed test, a specs/ directory for Markdown plans, and an .mcp.json that starts npx playwright run-test-mcp-server. The planner explores from the seed test and saves a plan, the generator executes each plan step live and writes a test from its log, and the healer runs failing tests and edits them. The documentation asks you to regenerate the definitions on every Playwright upgrade.

The planner and generator belong on the exploration side. Their plan and candidate spec go through the same review and mutants as anything else, and the planner's 1.64.1 tool list includes browser_run_code_unsafe, so the environment rules above apply to it.

The healer is different. Its definition at v1.64.1 lists fixing assertions and expected values among its remedies, tells it not to ask questions and to "do the most reasonable thing possible to pass the test", and lets it mark a test test.fixme() when it believes the application is broken. The documentation names the outcome: a passing test, or a skipped one. Pointed at the contract test while M1 is live, each of those moves turns the build green and none fixes the bug.

Use the healer on locators and waits after a redesign, on a branch, with a person reading the diff. A heal that changes an expected value or adds test.fixme is a product decision for whoever owns the behaviour, and no heal lands without a rerun of the mutants.

The price of keeping two tracks

Exploration costs tokens: every action returns a snapshot, and the Playwright MCP README now steers coding agents to playwright-cli with skills because tool schemas and accessibility trees are expensive context, keeping MCP for long, stateful exploratory loops. Review costs a person's time per candidate. Mutant runs cost wall-clock time, since every kill waits out a five-second assertion timeout; the 320 runs above took several minutes on a one-page app.

Skip the agent where a page object and a passing contract suite already exist. It earns its cost on a new flow, an unautomated page, or a flaky failure nobody can explain yet, where a session's trace and network log produce a hypothesis faster than reading code.

Where the series leaves you

Across eight parts, one rule kept coming back: a model may propose and explore, but the decision rests on something deterministic that someone can read. For servers that meant scripted clients and conformance runs; for authorization, tokens minted to be refused; for injection and SSRF, a scripted attacker and stub servers. For a browser it means a held response and a mutant that the test has to kill.

Apply this today

Pick one flow with a known or suspected timing bug. Start npx playwright mcp from your pinned @playwright/test with --browser chromium --isolated --caps=devtools,testing --save-session and an output directory, against an environment you own. Let the agent or the planner walk the flow with tracing on, and read the session log and trace for the order of requests. Write the held-route test for the dangerous order and a plain one for the ordinary order, then two or three mutants aimed at the root cause, and run the loop.

Keep the evidence in one pull request: session log, trace, a two-line root-cause note, the spec, and a mutant table with at least one 20-of-20 kill per mutant.

Questions about Playwright MCP and regression suites

Can a Playwright MCP agent run in CI as a smoke check?

It can run there, headless and isolated, and a pass means a model found some route through the flow on that build. That is too weak for a merge gate: another model version may take another route, and a failure may be the model rather than the product. Use it for triage on a failed build and keep the gate on the versioned suite.

Is it safe to commit the "Ran Playwright code" lines as a test?

As a starting point, yes. The emitted locators use roles and test ids, as you would. Rewrite the assertions, because a verification can be recorded weaker than the check the tool ran, and add the timing, data and starting state the session never controlled.

Primary references

• Playwright MCP README at v0.0.83. The CLI options (isolated, storage state, allowed origins with its not-a-boundary note, secrets, save session, output directory, settle timeout, caps), the user profile modes, the tool list per capability, the security note and the CLI-versus-MCP recommendation.
• npm registry entry for @playwright/mcp. Version 0.0.83 as latest, its publish date, and the pinned playwright 1.64.0-alpha dependency.
• Playwright MCP documentation: snapshots. The snapshot format, refs as targets, ref lifetime and the stale-ref error.
• Playwright MCP documentation: testing and assertions. The verify tools under --caps=testing and the statement that each records an expect(...) line for assembling a test.
• Playwright MCP documentation: tracing. browser_start_tracing and browser_stop_tracing under the devtools capability, and where traces are written.
• verify.ts in microsoft/playwright at v1.64.1. The text verification normalising its locator and recording a visibility assertion.
• MCP config.ts in microsoft/playwright at v1.64.1. The chrome channel assigned when no browser is named, chromium mapped to Chrome for Testing, and the 500 ms settle default.
• Playwright Test Agents guide. Planner, generator and healer, init-agents loops, seed tests, specs, regenerating on upgrade, and the healer's passing-or-skipped output.
• Healer agent definition at v1.64.1. Its instructions to fix assertions and expected values, not to ask questions, and to mark tests test.fixme().
• Playwright release notes. Test agents in 1.56, the trace CLI for agents in 1.59, and the bundled npx playwright mcp in 1.62.
• Playwright network guide. page.route, route.continue() and page.waitForResponse, used to hold and release the cart request.
• Playwright assertions reference. Auto-retrying assertions such as toHaveText and the five-second default timeout.
• Playwright Test command line. --repeat-each for the 20-run tallies.
• Playwright web server configuration. Starting the page under test from the config with its own environment.
• MCP security best practices. Local server compromise, and running local servers sandboxed with minimal default access.