Use AI for Performance Diagnosis Without Inventing Causes

A k6 gate fails with a p95 of 184 ms against a 100 ms objective. The dashboard export shows the database pool 90% busy, with requests queuing for a connection. Asked about it, a model proposes doubling the pool from 4 to 8. The change turns the gate green: p95 82 ms, exit code 0. The cause is still running. A pricing call slowed from 16 ms to 56 ms, and the service makes that call while it holds a database connection. The bigger pool hid the slowdown behind a passing threshold and doubled the connections the service may hold against PostgreSQL. Every number in that paragraph, including the model's proposal, came from the lab behind this article.

Earlier parts of this series allowed an assistant a narrow role: reading the aligned timeline when the load generator might be the bottleneck, summarising spans once load tests are joined to traces, and proposing which column saturated first in a capacity test. This final part builds the workflow behind that role: what goes into the bundle, how a reply is checked, and which experiment turns a candidate into a finding.

The run: one injected fault, one that nobody injected

The service, quote-api, answers GET /quote by borrowing a connection from a pool of four, running an 8 ms query, calling the pricing service, running a 4 ms query and releasing the connection. k6 sent 50 requests a second for 180 seconds, each delayed by a random 0 to 1 second so arrivals were not evenly spaced. Two changes were scripted. At 60 seconds a "deploy" moved the version label from 2.7.1 to 2.7.2 and changed nothing else. At 75 seconds the pricing stub's latency went from 15 ms to 55 ms. The pricing service is owned by another team in this story, so its latency appears on no dashboard the quote team reads.

The run also contained an episode that was not scripted. Rows from the 5-second evidence file, trimmed to the columns that matter:

t_s  version      client_p95_ms  pool_in_use_avg  pool_wait_p95_ms  db_query_p95_ms
  0  2.7.1                 34.8             1.54               1.0             11.0
 25  2.7.1                270.9             3.07             173.0             72.0
 55  2.7.1|2.7.2           35.9             1.51               1.0             11.0
 70  2.7.2                 34.2             1.80               1.0             11.0
 90  2.7.2                197.8             3.54             125.0             11.0
145  2.7.2                178.6             3.62             106.0             11.0

From 5 to 50 seconds, p95 climbed as high as 424 ms with the database query time rising alongside it. The lab did not diagnose it. The machine had restarted about 17 minutes earlier and its five-minute load average had recently been above 30, but no host or PostgreSQL metrics were recorded, so the evidence cannot say what happened. It stays in the record as unexplained, which is a legitimate result.

The 55 row hides a smaller trap. The deploy event is stamped at 60 seconds, yet the new version label appears in the row for 55 to 59, because the metrics writer labels each one-second bucket when it closes. One second of skew on one host is harmless; across hosts it adds to clock drift.

Build the bundle before anyone opens a chat

A bundle is the evidence frozen into files, with every decision about its contents made before a model reads it. Three decisions do most of the work.

Name the windows from the data. A person read the p95 column and wrote down episode_1 at 5 to 50 seconds, quiet at 55 to 75, and episode_2 from 75. Asked about the whole run at once, a model is invited to explain two episodes with one story.

Fix the trace selection rule in advance. This bundle took, per window, the five traces nearest the median duration and the five slowest, skipping the first 15 seconds of episode 2. A rule written down before anyone has a theory cannot be bent to fit one, and a reviewer can rerun it.

Write down what is missing. The manifest that travels with the files says so explicitly:

"not_included": [
  "pricing service metrics or logs (owned by another team)",
  "PostgreSQL server-side metrics",
  "load generator CPU"
]

A model cannot report a gap it was never told about, and whoever reads its answer needs to know which causes the bundle could never have shown.

Strip it, then prove the stripping worked

The metrics file is built from k6's CSV output and the service's per-second log, re-stamped as seconds since the load started. The raw sources around it are full of things that must not leave your network. The OpenTelemetry observability primer's own table of example span attributes includes client.address, url.query and user_agent.original. The lab's access log carried a bearer token, a session cookie and a customer email on every line. OWASP's 2025 entry on sensitive information disclosure (LLM02) recommends sanitising data before it reaches a model, and warns that instructions in the system prompt about what not to return may not be honoured. A prompt that says "ignore any secrets" is not a control. Removing them is.

// sanitize.mjs (excerpt): allowlist fields, pseudonymise accounts, then scan the output.
const salt = randomBytes(16); // kept in memory only: the bundle cannot be reversed to account ids
const pseudo = (v) => 'acct_' + createHmac('sha256', salt).update(v).digest('hex').slice(0, 10);

const KEEP = ['ts', 'traceId', 'status', 'dur_ms'];
for (const line of readFileSync(src, 'utf8').trim().split('\n')) {
  const rec = JSON.parse(line);
  const clean = Object.fromEntries(KEEP.map((k) => [k, rec[k]]));
  const url = new URL(rec.path, 'http://x');
  clean.route = url.pathname;                       // the query string carries the account id
  clean.account = pseudo(url.searchParams.get('account') ?? '');
  out.push(JSON.stringify(clean));
}

const patterns = {
  bearer: /bearer\s+\S+/gi, session_cookie: /session=/gi,
  email: /[\w.+-]+@[\w-]+\.[\w.]+/g, ipv4: /\b\d{1,3}(\.\d{1,3}){3}\b/g, raw_account: /acct-\d{4}/g,
};
// any hit exits non-zero and nothing is bundled

The allowlist decides what goes in, so a new header added next month is dropped by default. The deny scan proves what did not get through. Over the run's 9,001 access log lines:

records 9001
fields dropped {"method":9001,"path":9001,"client_ip":9001,"authorization":9001,"cookie":9001,"account_email":9001,"user_agent":9001}
leak scan {"bearer":0,"session_cookie":0,"email":0,"ipv4":0,"raw_account":0}

A scan that finds nothing has only proved something if it can find something. Run once against the raw file with the allowlist skipped, the same patterns reported 9,001 hits each for bearer, session cookie, email and IPv4, 18,002 for raw account ids (each line carries one in the path and one in the email), and the script exited 1. Keep that negative control with the bundle.

A reply format that code can check

Prose cannot be verified mechanically, so the lab asked for one JSON object in which every piece of evidence names its source precisely enough to look up:

Evidence items take one of three forms:
  {"file":"metrics-5s.csv","t_s":<row>,"column":"<column>","value":<number copied from that cell>}
  {"file":"traces.jsonl","trace_id":"<id>","span":"<span name>","dur_ms":<number copied>}
  {"file":"events.jsonl","t_s":<number>,"event":"<event>"}
Rules:
- Give at least three hypotheses, ranked. "against" lists evidence that weakens each one.
- Every value you cite must be copied from the files. Do not round, estimate or compute.
- Do not state that any hypothesis is the root cause. They are candidates until tested.
- "missing_evidence" lists what you would need that the bundle does not contain.

Each hypothesis also carries one experiment, changing one variable, with what it would show if the claim is right and if it is wrong. Requiring both predictions is the useful part. A hypothesis whose two outcomes look the same has no test.

A bundle that does not fit is cut, and nothing errors

With traces included, this bundle came to 6,657 prompt tokens. Ollama's FAQ, for 0.15.6 and for 0.40.9 alike, gives a default context window of 4,096 tokens. Sent without a num_ctx option, the request succeeded. The response reported prompt_eval_count 4096, exactly the window, and the reply was a JSON object describing a Python serialisation error in a file on a Windows desktop, unrelated to anything in the prompt. Part of the prompt had been dropped to fit, and the API returned no error.

Nothing reports this unless you check, so the lab's request wrapper now does:

const res = await fetch('http://127.0.0.1:11434/api/chat', {
  method: 'POST',
  body: JSON.stringify({
    model, stream: false, format: 'json', think: false,
    // Unset NUM_CTX sends no num_ctx, so Ollama uses its 4,096-token default.
    options: process.env.NUM_CTX ? { num_ctx: Number(process.env.NUM_CTX) } : {},
    messages: [{ role: 'user', content: contract }],
  }),
});
const j = await res.json();
// A prompt that filled the window was cut: the model never saw part of the bundle.
const ctx = Number(process.env.NUM_CTX ?? 4096);
if (j.prompt_eval_count >= ctx) throw new Error(`prompt filled the ${ctx}-token window; answer discarded`);

num_ctx=default prompt_eval_count=4096 eval_count=34
Error: prompt filled the 4096-token window; answer discarded

With num_ctx set to 16,384, the same bundle was read in full. With a hosted model, compare the input token count the provider reports with what you sent.

Check every citation before reading the argument

The checker loads the bundle and looks up each cited value at the place the reply says it came from:

// check-citations.mjs (excerpt)
if (e.file === 'metrics-5s.csv') {
  const row = metrics.get(Number(e.t_s));
  if (!row) return ['missing', `no row t_s=${e.t_s}`];
  if (!(e.column in row)) return ['missing', `no column ${e.column}`];
  if (typeof e.value !== 'number') return ['wrong', `cited ${JSON.stringify(e.value)}, not a number`];
  return e.value === Number(row[e.column])
    ? ['ok', '']
    : ['wrong', `t_s=${e.t_s} ${e.column}: cited ${e.value}, file has ${row[e.column]}`];
}

Exact equality is deliberate: the contract said copy, and a value that has been rounded or estimated is no longer evidence you can trace. The lab sent the metrics-only bundle three times and the full bundle three times. One of the six replies was not valid JSON, even though the request set format: "json" and Ollama's API reference says the output will always be a well-formed JSON object. Checked against the other five, here is the first full-bundle reply:

#1 The latency regression is caused by a high pool pending average, indicating that the syste
   0/1 citations verified
   WRONG: t_s=95 pool_pending_avg: cited 3.23, file has 2.50
#2 The latency regression is caused by a high db query time, indicating that the database is
   0/1 citations verified
   WRONG: t_s=95 db_query_p95_ms: cited 33, file has 11.0
#3 The latency regression is caused by a high server p95 ms, indicating that the server is ta
   0/1 citations verified
   WRONG: t_s=95 server_p95_ms: cited 35, file has 221.0
total: 0 ok, 3 wrong value, 0 not in bundle

Across the five checkable replies, 5 of 13 citations matched the cell they named, 7 named a cell holding a different value, and 1 pointed at nothing. The wrong values were not invented digits. In another reply, 190.7 was cited as the database query p95 at 95 seconds. That number is in the file, as client p95 at 120 seconds. Each wrong citation was a real number in the wrong place, which a reader skimming the prose would accept. OWASP's 2025 entry on misinformation (LLM09) names that reader's failure as overreliance, and its mitigation is cross-checking against trusted sources. Here the trusted source is the bundle, and the cross-check is code.

The checker also exposes what the replies did not do. None of the six cited a trace, although the full bundle held thirty. None mentioned the pricing call. Every full-bundle reply stated its first hypothesis as "is caused by", which the contract forbade. Two of the metrics-only replies ranked the deploy first.

Llama 3.2 3B is a small model, and this lab did not test a larger one. The checker does not care which model wrote the reply, and it runs in well under a second. Run it on every answer, and drop any hypothesis without a verified citation before a person reads its reasoning.

What the bundle could already say without a model

The decisive signal in this bundle takes one division, which the contract deliberately left out of the citations. By Little's Law, as part 6 used it, connections in use equal throughput multiplied by hold time, so the average hold time is the pool's in-use figure divided by served requests per second.

$ awk -F, 'NR>1 && $3>40 {printf "%3d  use_ms=%5.1f  db_query_p95=%s  wait_p95=%s\n", $1, $7/$3*1000, $10, $9}' metrics-5s.csv
  0  use_ms= 31.0  db_query_p95=11.0  wait_p95=1.0
 30  use_ms= 75.0  db_query_p95=84.0  wait_p95=312.0
 60  use_ms= 30.8  db_query_p95=11.0  wait_p95=1.0
 75  use_ms= 70.8  db_query_p95=11.0  wait_p95=76.0
 95  use_ms= 71.3  db_query_p95=11.0  wait_p95=151.0

Both episodes show requests waiting for a connection. Their hold times rose for different reasons. In episode 1, the hold time rose together with query time. In episode 2, hold time more than doubled, from 31 ms to 71 ms, while query p95 stayed at 11 ms. Something other than the database was being done while the connection was out. The spans recorded for every request say what: in the quiet window the pricing span averaged 15.7 ms, and in episode 2 it averaged 55.8 ms. The episode 2 spans add up: 10.3 ms and 5.0 ms of queries plus 55.8 ms of pricing make 71.1 ms, the measured hold time.

OpenTelemetry's database client conventions name both quantities: db.client.connection.wait_time, the time to obtain a connection, and db.client.connection.use_time, the time between borrowing a connection and returning it. Wait time rising with use time flat means the pool is too small for the load. Use time rising while operation time stays flat means work is being done under a held connection. Those connection pool metrics are still at Development status in the conventions, while db.client.operation.duration is Stable, so check whether your instrumentation emits them before you plan on reading them.

That is still a hypothesis, but now one with cited telemetry and a prediction the pool hypothesis does not share.

Experiments that change one thing

Three candidates came out of the evidence and the model replies: the deploy, a pool too small for the traffic, and the pricing call held inside the connection. Each run lasted 90 seconds at 50 requests a second with the same jitter; figures cover 10 to 85 seconds. B, E1 and E3 each ran twice, the repeats moved p95 by under 2 ms, and the first run is shown.

Run B rejects the deploy: it reproduces the incident on 2.7.1 with no deploy at all, as the quiet rows on 2.7.2 had already suggested.

The pool hypothesis is the instructive one. Doubling the pool removed the queue and cut p95 by 55%, from 183.7 ms to 82.2 ms, which passes the 100 ms gate. Judged on p95 alone, it looks confirmed. Its own prediction was different: if the pool were simply too small, hold time would stay near 31 ms and p95 would return to the 36 ms baseline. Hold time stayed at 71 ms, and p95 at more than twice the baseline. Pool size amplified the incident; it did not cause it.

Runs A and B differ in exactly one variable, pricing latency, and that variable alone moved p95 from 36.3 ms to 183.7 ms. That confirms the pricing slowdown as the driver. Run E3 tests the mechanism: with the call moved before the connection is borrowed, hold time fell to 15.3 ms, the two queries alone, and the pool sat at 19% busy. E1 and E3 have the same p95, so p95 could never have chosen between them; E1 holds 3.57 connections to serve the load and E3 holds 0.76. The remaining 45 ms above baseline is the pricing call itself, a finding for the team that owns it.

The SRE book's chapter on troubleshooting puts the design rule plainly: an ideal test has mutually exclusive alternatives, ruling one group of hypotheses in and another out. Choose the measurement before the run by asking where the hypotheses' predictions differ. Here that was hold time, not latency. In production you cannot set another team's latency, but a fault-injection delay on the dependency in staging gives the same one-variable comparison.

The record that closes the investigation

run: quote-api, 50 req/s, pool 4, k6 v2.3.0, bundle built 2026-10-23
hypotheses:
  - claim: deploy 2.7.2 slowed quote-api
    evidence: deploy event t=60s; quiet rows t=60-70s on 2.7.2 (p95 34-43 ms)
    experiment: incident condition on 2.7.1, no deploy (run B)
    result: p95 183.7 ms without the deploy
    status: rejected
  - claim: pool of 4 is too small for 50 req/s
    evidence: pool_in_use_avg 3.54 at t=90s; pool_wait_p95_ms 151 at t=95s
    experiment: pool 4 -> 8, nothing else (run E1, twice)
    result: queue gone, p95 82.2 ms; hold time unchanged at 71.4 ms
    status: rejected as cause; amplifier
  - claim: slower pricing call made while holding a connection
    evidence: hold time 31.0 -> 70.8 ms at t=75s with db_query_p95 flat at 11 ms;
              pricing span mean 15.7 ms (quiet) vs 55.8 ms (episode 2)
    experiment: pricing latency as the only change (A vs B); call moved before borrow (E3, twice)
    result: p95 36.3 -> 183.7 ms; E3 hold time 15.3 ms, pool 19% busy
    status: confirmed; residual +45 ms owned by pricing
  - claim: episode 1, t=5-50s, query time and p95 elevated
    evidence: db_query_p95_ms up to 93 ms; pricing span unchanged
    missing: host CPU, PostgreSQL server metrics
    status: unexplained

The rejected entries are worth as much as the confirmed one. Someone will propose the pool change at the next incident, and the record already holds its test and result. The SRE book calls negative results conclusive, and argues for publishing them so the next team does not rerun the experiment.

Where AI-assisted diagnosis goes wrong

One story for two episodes. The symptom is a single hypothesis citing rows from both ends of the run. Both episodes here showed pool waiting, and only hold time against query time told them apart. Reject any citation that falls outside the window its claim is about.

The marker on the timeline. The symptom is a deploy or config change ranked first because it came shortly before the onset. The SRE chapter lists hunting spurious correlations among the common troubleshooting pitfalls, and a timeline annotation is the easiest one to find. Look at the rows between the marker and the onset: here, 15 seconds of normal latency on the new version. Then rerun the fault without the change.

A remedy accepted because the gate went green. The symptom is an experiment judged only on the metric that failed, as E1 would have been. Require each experiment to predict a mechanism metric as well as the latency.

Two variables in one run. The symptom is a fix that bundles a pool change with moving the pricing call. Each remedy produced 82 ms on its own, so a combined run would have credited both and taught nothing about either. Since active tests can change later results, as the SRE book warns, revert each change before the next and log every configuration you ran.

Run this on the next regression

Take the last performance regression your team investigated and rebuild its evidence as a bundle: one clock, windows named from the data, a fixed trace selection rule and a manifest of what is missing. Run an allowlist sanitiser over it, then run the scan against the raw files to prove it can see a leak. Write the reply contract with copied values and two predictions per experiment, set the model's context window explicitly, and reject any reply whose prompt filled it. Check every citation with code. For the top two hypotheses, find the metric where their predictions differ, then run one experiment per hypothesis and repeat the deciding run. Keep the bundle, the checker output, both runs and the hypothesis record together, including the hypotheses you rejected and anything left unexplained.

Questions about AI-assisted performance diagnosis

Can the bundle go to a hosted model instead of a local one?

If it passes the scan and your data policy allows it. The contract, the context check and the citation checker apply unchanged; only the guarantee that nothing leaves the machine is lost, which makes the sanitiser and its negative control matter more.

Is a metrics-only bundle worth sending?

It can rank candidates, and here the hold-time division was available from metrics alone. It cannot show what happened inside the hold. Two of the three metrics-only replies ranked the deploy first, and none could have named the pricing call. Say in the manifest that traces are missing, so the reply has to list them as missing evidence.

Where does this sit next to thresholds and canary rules?

After them. Thresholds decide whether a run failed, validity gates decide whether it measured anything, and canary stop rules decide whether a release continues. Diagnosis starts once one of them has fired, and a model's answer changes none of them.

Primary sources for this part

• Effective Troubleshooting, Google SRE book chapter 12. The hypothetico-deductive process, pitfalls including spurious correlations and latching onto past causes, tests with mutually exclusive alternatives, side effects of active tests, taking notes, and negative results.
• Semantic conventions for database client metrics. db.client.connection.wait_time and db.client.connection.use_time at Development status, and db.client.operation.duration as Stable.
• OpenTelemetry observability primer. Signals that must be emitted for questions to be answerable, SLIs from the user's perspective, and the example span attributes that carry client addresses, queries and user agents.
• OWASP Top 10 for LLM Applications 2025, LLM02 Sensitive Information Disclosure. Sanitising data before it reaches a model, and system prompt restrictions that may not be honoured.
• OWASP Top 10 for LLM Applications 2025, LLM09 Misinformation. Plausible but unfounded output, overreliance, and cross-verification with human oversight.
• Ollama FAQ at v0.15.6. The 4,096-token default context window and the num_ctx option; the v0.40.9 FAQ gives the same default, and the v0.40.9 release was the newest on the check date.
• Ollama API reference at v0.15.6. The format parameter and its statement that JSON mode returns well-formed JSON.
• node-postgres Pool API. Requests waiting in a FIFO queue when every client is checked out, and the pool's count properties.
• Grafana k6 thresholds. The pass and fail criteria behind the 100 ms gate, and the exit status a failed threshold produces.
• k6 constant-arrival-rate executor. Iterations started independently of response time, the open model the lab used.
• k6 CSV output. The per-request rows and timestamps the bundle was built from.
• grafana/k6 v2.3.0 release. The version that produced every run here.