Test Capacity Limits Before Production Finds Them

A capacity report says the statement API sustained 110 requests a second with zero errors, and the CPU graph hovered around 40%. Both statements are true of a run described below. So is the rest of it: the service completed 71 of those 110 requests each second, the other 39 joined a queue for a database connection, and the requests sent during that step had a 95th percentile of 17.5 seconds. Nothing failed because nothing was allowed to fail. The connection pool's default is to wait indefinitely, so the first limit, four database connections, was invisible to both the error rate and the CPU graph.

Part 1 of this series used a breakpoint ramp to find where a service starts to fail. This part finds which resource makes it fail, proves that attribution with a controlled rerun, and checks that the service comes back afterwards.

List what can run out before choosing a load shape

The service under test has one endpoint, GET /statement. Each request borrows a connection from a node-postgres pool, runs a 50 ms query, releases the connection, then builds and gzips an 88 KB statement, which costs about 3.5 ms of CPU on the single Node.js event loop. The query is SELECT pg_sleep(0.05), a stand-in whose duration does not change with load. PostgreSQL documents that the sleep lasts at least as long as requested. A real query makes no such promise, which matters later.

Before any traffic, write down every bounded resource on that path, its configured limit, what saturation looks like, and what happens to the request that arrives after it:

Then predict the order. For a pool, Little's Law gives the ceiling: connections in use equal completions per second multiplied by how long each completion holds a connection. Four connections each held for the 51.7 ms a request takes at low load run out at 4 ÷ 0.0517 ≈ 77 requests a second. That is a hypothesis for the run to confirm or refute, and it names the column that should move first. The same relationship sizes virtual users in the guide to building a k6 load model from traffic assumptions.

A stepped ramp that labels every request with its step

The test raises one dimension, the arrival rate, and nothing else. Replica count, pool size, payload and data stay fixed, and autoscaling is off. Each level holds for 18 seconds after a 2-second ramp, so the queues settle and each level yields one comparable row. A final 40 seconds at 50 requests a second measures recovery:

// capacity.js
import http from 'k6/http';
import { sleep } from 'k6';
import exec from 'k6/execution';

const BASE = __ENV.BASE_URL || 'http://127.0.0.1:3366';
const STEPS = (__ENV.STEPS || '20,40,60,70,80,90,110').split(',').map(Number); // requests per second
const HOLD = 20; // seconds per step, the first 2 of them ramping
const RECOVERY_RATE = 50;
const JITTER = Number(__ENV.JITTER || 0); // seconds; 0 keeps k6's evenly spaced starts

const stages = [];
for (const rate of STEPS) {
  stages.push({ target: rate, duration: '2s' }, { target: rate, duration: `${HOLD - 2}s` });
}
stages.push({ target: RECOVERY_RATE, duration: '1s' }, { target: RECOVERY_RATE, duration: '39s' });

const RAMP_END = STEPS.length * HOLD;

// one tag value per step, so every metric can be read per load level
function stepAt(seconds) {
  if (seconds < RAMP_END) return `r${String(STEPS[Math.floor(seconds / HOLD)]).padStart(3, '0')}`;
  return seconds < RAMP_END + 5 ? 'drain' : 'recovery';
}

const thresholds = { dropped_iterations: ['count==0'] };
for (const step of [...STEPS.map((_, i) => stepAt(i * HOLD)), 'drain', 'recovery']) {
  thresholds[`http_req_duration{step:${step}}`] = ['p(95)<250'];
  thresholds[`http_req_failed{step:${step}}`] = ['rate<0.01'];
}

export const options = {
  scenarios: {
    capacity: {
      executor: 'ramping-arrival-rate',
      startRate: STEPS[0],
      timeUnit: '1s',
      preAllocatedVUs: Number(__ENV.PREALLOCATED_VUS || 50),
      maxVUs: 1500,
      stages,
    },
  },
  thresholds,
  discardResponseBodies: true,
  summaryTrendStats: ['med', 'p(95)', 'max'],
};

export default function () {
  const step = stepAt(exec.instance.currentTestRunDuration / 1000);
  if (JITTER) sleep(Math.random() * JITTER);
  http.get(`${BASE}/statement?account=${exec.scenario.iterationInTest % 5000}`, {
    headers: { 'X-Load-Step': step },
    tags: { name: 'GET /statement', step },
  });
}

k6 turns tagged requests into sub-metrics that thresholds can select, so the end-of-test summary prints a latency line and a failure line per step. The objective, p95 under 250 ms and failures under 1%, marks the step where the service leaves it; this run measures rather than gates, so exit code 99 is expected. The same label travels in an X-Load-Step header, so the server buckets its own figures by step with no clock alignment. And dropped_iterations must stay at zero, or the generator did not deliver the load a row claims; debugging k6 results when the load generator is the bottleneck covers that case.

The first version of the script had no name tag. With 5,000 account ids in the URL, k6 v2.3.0 warned before the run finished:

level=warning msg="The test has generated metrics with 100015 unique time series, which is higher than the suggested limit of 100000 and could cause high memory usage. Consider not using high-cardinality values like unique IDs as metric tags or, if you need them in the URL, use the name metric tag or URL grouping."

On the server side, the pool is configured once, and the bounded variant refuses work before it joins a long queue:

// statement-api.mjs (excerpt)
const POOL_MAX = Number(process.env.POOL_MAX ?? 4);
const SHED = process.env.SHED !== 'off';
const QUEUE_MAX = Number(process.env.QUEUE_MAX ?? 8);
const ACQUIRE_TIMEOUT_MS = Number(process.env.ACQUIRE_TIMEOUT_MS ?? 200);

const pool = new pg.Pool({
  database: 'postgres',
  max: POOL_MAX,
  // 0 (the default) means a request waits for a connection indefinitely
  connectionTimeoutMillis: SHED ? ACQUIRE_TIMEOUT_MS : 0,
});

function shed(res, b, reason) {
  b[reason]++;
  res.writeHead(503, { 'retry-after': '1', 'content-type': 'application/json' });
  res.end(JSON.stringify({ error: 'overloaded', reason }));
}

// inside the request handler, after the step bucket b is chosen
if (SHED && pool.waitingCount >= QUEUE_MAX) return shed(res, b, 'shedQueue');
const t0 = performance.now();
let client;
try {
  client = await pool.connect();
} catch {
  return shed(res, b, 'shedTimeout');
}
const t1 = performance.now(); // t1 - t0 is the wait for a connection

Utilisation needs care. A sampler on a timer runs on the same event loop as the work, so it fires at biased moments; an early version of this server reported a utilisation below what Little's Law allowed at the same throughput. The pool's own acquire and release events give an exact time-weighted figure instead:

// time-weighted count of checked-out connections, from the pool's own events
let checkedOut = 0;
let lastChange = performance.now();
function accrue() {
  const now = performance.now();
  const b = buckets.get(currentStep);
  if (b) b.checkedOutMs += checkedOut * (now - lastChange);
  lastChange = now;
}
pool.on('acquire', () => { accrue(); checkedOut++; });
pool.on('release', () => { accrue(); checkedOut--; });

Reading the saturation table

With four connections, a wait queue capped at eight and a 200 ms acquire timeout, k6 run capacity.js produced this table. Served rate, pool, wait and CPU figures come from the server's per-step buckets, and latency and failures from k6's per-step sub-metrics:

Served is slightly below offered at the low steps because each step includes its 2-second ramp: at the 70 step, 2 seconds averaging 65 and 18 seconds at 70 give 1,390 requests, 69.5 a second. Every failure in the table is a deliberate 503 from the bounded queue.

Between 70 and 80, four columns move together. The served rate stops at about 71 a second, the pool reaches 100%, the queue sits at its cap of eight, and the wait for a connection jumps from 6.5 ms to 112 ms. Two columns stay flat. Query time is 52 ms at every step, so the database answered as quickly at 110 offered as at 20; it is not the bottleneck. CPU stays near 35%. The limit is the pool in front of the database, and it bounds this service to about 71 served requests a second.

Little's Law also explains the gap between the prediction and the plateau. Below the knee, connections checked out divided by throughput gives the hold time per request: 0.899 × 4 = 3.60 connections at 69.5 a second is 51.7 ms, the query time. At the plateau, 4 ÷ 71.2 gives 56.2 ms. Once requests are waiting, a released connection is handed straight to the next waiter, but in Node.js that waiter's code resumes on the same event loop, after the releasing request has rendered and gzipped its response. Moving the render before pool.connect(), and changing nothing else, raised the plateau at 110 offered from 71 to 75.9 a second. Take the ceiling from measured throughput, not from the query timer. The same law predicts the wait: eight waiters ahead, served at 4 connections per 56.2 ms, is 8 × 56.2 ÷ 4 = 112.4 ms, which is the measured p95.

Two averages that hid this limit

The pool was 89.9% checked out at 70 a second with little waiting, which looks like headroom. Part of the reason is the generator. k6's VU allocation guide states that it starts the configured number of iterations evenly across the time unit, so at 70 a second a request arrives every 14.3 ms. Real arrivals cluster. Running the same steps with JITTER=1 delays each request by a random 0 to 1 second without changing the average rate:

Under the 1% failure budget, the highest passing step fell from 70 to 60. Held at a constant 65 a second for 30 seconds, the wait-for-connection p95 was 0.1 ms with even spacing and 89.4 ms with jitter, and the jittered run shed 15 of 1,950 requests. A uniform random delay is not a model of your traffic; it shows how sensitive the limit is to burstiness. Your own per-second arrival counts say how bursty production is.

The second average is CPU. It sat between 35% and 36% while the pool was saturated. Suppose three replicas, each requesting one CPU, ran under a HorizontalPodAutoscaler targeting 70% CPU. Its documented formula, ceil(currentReplicas × currentValue ÷ desiredValue), gives ceil(3 × 36 ÷ 70) = ceil(1.54) = 2: remove a replica, and a third of the connections, while every pool has eight requests waiting. The default five-minute downscale stabilisation window delays that decision without correcting it. When a pool is the first limit, scale on its wait or queue length.

Zero errors and a 17-second p95: the unbounded queue

The opening report came from the same service with SHED=off, which leaves node-postgres at its default of waiting indefinitely. The steps and pool are unchanged:

Both versions served the same 71 requests a second; the unbounded one recorded 71.3, 71.4 and 71.1 at those three steps. Refusing work did not cost throughput; it changed what happened to the excess. The Google SRE book states the same goal: under overload, a well-designed component rejects some requests and survives, without significantly reducing the rate at which it serves the rest. No unbounded request failed because k6's default request timeout is 60 seconds and the longest wait was about 18.

The backlog follows from the arithmetic. When the overload ended, 1,282 requests were queued; at 71 a second that is 18.1 seconds of waiting, and k6 measured a p95 of 18.24 seconds for requests that arrived then. During recovery, 50 arrived and 71 were served each second, so the queue shrank by about 21 a second: 1,282 ÷ 21 ≈ 61 seconds to drain, longer than the 40-second recovery window. When k6 stopped sending, 459 requests were still queued, and the queue emptied seven seconds later. The bounded version's queue was empty within two seconds of the drop: 8 waiting at one per-second sample, none at the next.

Holding those waiting requests took k6 1,297 VUs. A first attempt with 900 pre-allocated dropped 203 iterations, failed the dropped_iterations threshold, and was discarded.

Pick the shed's status code deliberately. RFC 9110 defines 503 as a server temporarily unable to cope because of overload, optionally with Retry-After. RFC 6585 defines 429 as a client that has sent too many requests in a given time: rate limiting. A capacity shed is about the server, so 503 fits, and keeping 429 for per-client limits lets a report tell them apart. k6 counts both as failed, because only 200 to 399 are expected by default, so classify failures by status before calling them defects.

The queue's size sets the worst wait. The SRE book suggests that for fairly steady traffic a queue of half the worker pool or less is usually better, because it rejects early. This lab's queue of 8 was twice the pool and cost up to 112 ms. A queue of 2 would cap the wait near 2 × 56.2 ÷ 4 ≈ 28 ms and shed sooner; bursty traffic can justify more.

Change only the suspect and run the same steps

A hypothesis about the first limit stands only when changing that resource alone moves the plateau. With POOL_MAX=8 and the identical script, the 110 step served 108.9 a second with no failures, a p95 of 59.47 ms and CPU at 49.3%. Extending the steps to 210 found the new plateau at 142.7 to 144.0 served a second, 2.0 times the four-connection figure, with the pool at 100%, eight waiting, and CPU between 64% and 67%. The pool was still the first limit, at twice the size.

That extended run also shed 11 requests at the 110 step and 37 at 130, with the pool 71% and 84% checked out. An earlier run of the same configuration shed none at those steps. One run is a sample. Repeat the decisive steps before writing down a limit.

A bigger pool moves connections onto PostgreSQL. With the default 100 max_connections and 3 superuser slots, and reserved_connections at its default of 0, ordinary roles have 97 slots. Twelve replicas with pools of 8 can hold 96. A thirteenth gets one more connection, and its next is refused with the PostgreSQL 18 source's message remaining connection slots are reserved for roles with the SUPERUSER attribute. The node-postgres sizing guide likewise says to add up max across every instance. The lab's query also costs the database almost nothing; a real one competes for its CPU, locks and I/O, so a rerun after enlarging a pool must watch database-side waits. The limit may simply move into the database.

An AI assistant is useful at this stage for reading a per-step table and proposing which column moved first. Its hypothesis is a candidate. The 2.0 times move of the plateau when only the pool changed is the evidence.

When the curve points at the wrong resource

A plateau with no saturated resource: throughput flattens, but no listed queue grows. The bottleneck is off the list, most often the generator or its link. Check generator CPU and received bytes per step before touching the service.

A plateau that moves between runs: something besides load changed, such as replica count, an autoscaler left on, cache state, payload size or data volume. Here moving a 3.5 ms render moved the ceiling from 71 to 75.9. A throughput figure is portable only with the configuration that produced it, so record both.

Latency rising with no errors and no visible queue: the waiting happens where no dashboard counts, such as a client library's own pool, a lock or an in-process queue. Time the wait at each borrow point, as t1 - t0 does above.

The record a capacity test should leave

service:       statement-api, 1 replica, pg pool max 4, wait queue 8, acquire timeout 200 ms
workload:      GET /statement, 88 KB gzip response, 50 ms query, autoscaling off
first limit:   database connection pool, 71 served/s
evidence:      pool 100% checked out, 8 waiting, wait p95 112 ms;
               query p95 flat at 52 ms; CPU 35-36%
confirmed by:  pool 4 -> 8 alone moved the plateau to 143/s (2.0x)
safe limit:    60/s with jittered arrivals (p95 121 ms, 0 failures)
overload:      503 + Retry-After; served rate held at 71/s
recovery:      queue empty within 2 s of dropping to 50/s
database:      replicas x pool max <= 97 slots
next limit:    not yet found; rerun watching PostgreSQL waits and CPU

Turning the safe limit into a per-release check is the job of thresholds that fail CI for the right reasons. The k6 Script Generator gives a reviewable starting script to add step tags to, and the HTTP Status Code Reference helps when sorting a run's failures into sheds, rate limits and defects.

Apply this today

Pick one service and write its resource table: every pool, queue and slot limit on the request path, with its configured size and what it does when full. Look for wait-forever defaults such as connectionTimeoutMillis: 0; each one is a place where overload will look like slowness instead of errors. Expose the wait count and the time to acquire for each pool, per instance.

Then run a stepped arrival-rate ramp with a step tag on every request and autoscaling off, read the per-step table for the column that saturates while served throughput flattens, and run the decisive steps twice. Rerun with only that resource changed. Keep the two step tables, the confirming rerun and the recovery trace; together they are the evidence for the capacity figure.

Questions about capacity testing

Should the capacity test run on every pull request?

No. A stepped ramp needs a dedicated environment and is designed to fail its upper steps. Run it when the request path, pool settings, payload or dependencies change, and on pull requests run a short constant-rate check at the recorded safe limit.

How large should a wait queue be?

Start from the worst wait you can accept. The longest wait is roughly queue length × hold time ÷ pool size, so a 100 ms budget with 4 connections held 56 ms each allows a queue of about 7. The SRE book's guidance of half the pool or less suits steady traffic; size it from observed bursts if traffic is spiky.

Is a bigger pool the fix once the pool is the limit?

Only if the database has room for the extra concurrent work and the slot budget allows it. The node-postgres guide suggests that an application starved for connections look at making its queries faster, or at caching, before raising the pool. A faster query shortens the hold time, which raises the ceiling without asking more of the database.

Why steps instead of a continuous ramp?

A continuous ramp never lets a queue settle, so every moment mixes the previous rate's build-up with the current one. Steps give each level a steady state and its own row. A breakpoint ramp is still the quick way to find roughly where to put the steps.

Primary references

• ramping-arrival-rate in the k6 executor docs: staged target rates, and iteration starts spaced fractionally across each second.
• Arrival-rate VU allocation (k6 manual): iterations started evenly across the time unit, the median duration × rate estimate, and dropped_iterations when no VU is free.
• Setting thresholds on tagged sub-metrics in k6: the metric{tag:value} selector used for one threshold per step.
• The k6/execution module: instance.currentTestRunDuration and scenario.iterationInTest.
• Request params for k6 HTTP calls: per-request tags and the 60-second default timeout.
• setResponseCallback in k6/http: status codes 200 to 399 counted as expected by default, which decides http_req_failed.
• Tags, groups and URL grouping in k6: setting tags.name for dynamic URLs.
• k6 release v2.3.0 on GitHub: the version every run here used.
• node-postgres Pool API: max defaulting to 10, connectionTimeoutMillis defaulting to 0 with no timeout, waitingCount, and the acquire and release events.
• node-postgres pool sizing guide: summing max across every instance, and looking at query speed before raising the pool.
• PostgreSQL 18 connection settings: max_connections defaulting to 100, superuser_reserved_connections to 3 and reserved_connections to 0.
• postinit.c at REL_18_6: the error text when only superuser slots remain.
• PostgreSQL date/time functions, delaying execution: pg_sleep lasting at least as long as requested.
• PostgreSQL versioning policy: 18.6 as the current minor release and PostgreSQL 14's end of life on 2026-11-12.
• RFC 9110, HTTP Semantics: 503 for temporary overload, and Retry-After with a 503.
• RFC 6585, Additional HTTP Status Codes: 429 for a client sending too many requests.
• Addressing Cascading Failures, Google SRE book: small queues relative to the pool, shedding with 503, testing beyond the breaking point and back to nominal load.
• Kubernetes HorizontalPodAutoscaler: the replica formula, the 0.1 tolerance and the five-minute downscale stabilisation default.
• Node.js releases page: 26.12.0 as the latest release on the check date.