Connect Load Tests to OpenTelemetry Traces

A checkout load test fails its gate in the steady phase: 2% of payments returned 503. The latency percentile tells you users waited. It does not tell you where the time went, and when you open the tracing backend for that hour you find tens of thousands of checkout traces from the load test, a synthetic monitor and two other teams sharing the environment. Without a way to select the traffic your run sent, the investigation turns into comparing broad time windows and guessing which spans belong to the experiment.

Part 2 of this series turned objectives into k6 thresholds that fail CI for the right reasons, and Part 4 split the workload into one scenario per journey. This part connects a failing threshold to the spans and log lines behind it, for the same bounded set of requests, without creating a metric series per user or leaking test context to a payment provider.

What the tracing library actually sends

One call in the init context replaces the request functions of the k6/http module with traced versions. It does not replace http.batch, so requests sent in a batch carry no trace context:

import tempo from 'https://jslib.k6.io/http-instrumentation-tempo/1.0.1/index.js';

tempo.instrumentHTTP({ propagator: 'w3c' }); // the default; 'jaeger' sends uber-trace-id instead

The library's source shows four behaviours worth knowing before you query anything, and the test run confirmed each of them.

Every request is its own trace. The library generates a new trace ID inside each request() call. The checkout journey below sends two requests per iteration, and the run's 2,000 requests carried 2,000 distinct trace IDs. A journey is not a trace. You find one request by its trace ID and the whole journey by its context attributes, which is why the next section exists.

The IDs are recognisable. Each trace ID starts with the fixed prefix dc0718, followed by bytes derived from the clock and then random bytes. All 2,000 IDs in the run started that way, which lets you recognise k6-originated traces at a glance.

The parent span never arrives. The parent-id field in traceparent is 16 random hex characters, and k6 exports no span with that ID. The first service's server span is therefore the top of the recorded trace, pointing at a parent nobody reports. That is expected, not a broken pipeline.

The trace ID goes into metadata, not tags. The library sets exec.vu.metrics.metadata.trace_id for the duration of the call. The k6/execution reference describes metadata as similar to tags but usable for high-cardinality data, unavailable to thresholds, and "likely handled differently by each output". With --out json, each data point carries it:

{"metric":"http_req_duration","type":"Point","data":{"time":"2026-10-19T16:51:01.285354+05:30","value":963.226,"tags":{"error_code":"1503","expected_response":"false","group":"","journey":"checkout","method":"POST","name":"checkout","phase":"steady","proto":"HTTP/1.1","scenario":"steady","status":"503","test.run":"perf-2026-10-24-a","url":"checkout"},"metadata":{"trace_id":"dc071818dfea5ca3c16c9422504d92dd"}}}

The CSV output writes it into a trailing metadata column as trace_id=dc0718…. The end-of-test summary aggregates, so it shows no trace IDs at all; keep a per-request output file whenever you intend to follow a slow request into a trace.

Three keys whose values you can list

The context that selects your traffic needs values you could write down before the test starts:

• test.run: one value per run, supplied by CI so it matches the job that produced it, for example perf-2026-10-24-a.
• journey: one of the scenarios from your workload model, such as checkout or browse.
• phase: ramp, steady or recovery, matching how the test is built.

The script below carries all three. Phases are separate scenarios, each with a phase tag, so every metric the scenario emits is labelled without extra code. The run ID comes in through -e and is validated, because a test with a missing or malformed ID produces data nobody can select later:

// checkout.js: k6 run -e TEST_RUN=perf-2026-10-24-a checkout.js
import http from 'k6/http';
import exec from 'k6/execution';
import { check } from 'k6';
import tempo from './tempo.js';

const BASE = __ENV.BASE_URL || 'http://127.0.0.1:3581';
const RUN = __ENV.TEST_RUN;
if (!/^perf-\d{4}-\d{2}-\d{2}-[a-z0-9]+$/.test(RUN || '')) {
  throw new Error('set TEST_RUN, e.g. -e TEST_RUN=perf-2026-10-24-a');
}

tempo.instrumentHTTP({ propagator: 'w3c', sampling: Number(__ENV.SAMPLING || 1) });

export const options = {
  tags: { 'test.run': RUN, journey: 'checkout' },
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',
      exec: 'checkout',
      startRate: 0,
      timeUnit: '1s',
      stages: [{ target: 20, duration: '20s' }],
      preAllocatedVUs: 20,
      tags: { phase: 'ramp' },
    },
    steady: {
      executor: 'constant-arrival-rate',
      exec: 'checkout',
      rate: 20,
      timeUnit: '1s',
      startTime: '20s',
      duration: '40s',
      preAllocatedVUs: 30,
      tags: { phase: 'steady' },
    },
  },
  thresholds: {
    'http_req_duration{phase:steady,name:checkout}': ['p(95)<300'],
    'http_req_failed{phase:steady,name:checkout}': ['rate<0.01'],
  },
};

export function checkout() {
  const t = exec.vu.metrics.tags;
  const headers = {
    'Content-Type': 'application/json',
    baggage: `test.run=${RUN},journey=checkout,phase=${t.phase}`,
  };
  http.post(`${BASE}/cart/items`, '{"sku":"SKU-1"}', { headers, tags: { name: 'cart' } });
  const res = http.post(`${BASE}/checkout`, '{}', { headers, tags: { name: 'checkout' } });
  check(res, { 'order placed': (r) => r.status === 201 });
}

The import points at a local copy of the 1.0.1 file from the jslib repository, so the run did not depend on the network; on the day of the check it was byte-for-byte identical to the file served at the jslib.k6.io URL in the first snippet. Two details are easy to get wrong.

The baggage header is the W3C format for passing key-value context between services: comma-separated key=value members, with keys restricted to the HTTP token characters, which include the dot. The specification only obliges a platform to propagate all members while there are at most 64 of them and the header stays within 8,192 bytes. Three short keys are far below that, which is one more reason to keep the context small.

The phase comes from exec.vu.metrics.tags, which includes the scenario's own tags. Logged from inside a scenario tagged phase: 'steady', it printed {"scenario":"a","phase":"steady","group":""}. If your phases are stages of a single ramping scenario rather than separate scenarios, the k6 tags documentation shows tagWithCurrentStageIndex from the utils jslib, which adds a numeric stage tag to every sample in the iteration. You then keep a mapping from stage number to phase name next to the script.

An identifier as a tag multiplies the series

The tempting mistake is to tag requests with whatever identifies them: an account, a cart, an order. Here is the same script fragment both ways, run for ten seconds at 20 iterations per second:

// cardinality.js: the same identifier as a tag, then as metadata
import http from 'k6/http';
import exec from 'k6/execution';

export const options = {
  scenarios: {
    c: { executor: 'constant-arrival-rate', rate: 20, timeUnit: '1s', duration: '10s', preAllocatedVUs: 10 },
  },
};

export default function () {
  const cart = `cart-${exec.scenario.iterationInTest}`;
  if (__ENV.CART_AS === 'tag') exec.vu.metrics.tags.cart = cart;          // wrong
  else exec.vu.metrics.metadata.cart = cart;                              // right
  http.post('http://127.0.0.1:3581/cart/items', '{}', { tags: { name: 'cart' } });
}

Counting distinct tag sets per metric in each JSON output:

CART_AS=tag
http_req_duration  200
http_reqs          200
iterations         200
CART_AS=metadata
http_req_duration  1
http_reqs          1
iterations         1

Two hundred requests became two hundred series of one sample each. Locally that is only a bigger file. Sent onward, it is a cost: the k6 OpenTelemetry output (--out opentelemetry) maps all k6 tags to OpenTelemetry attributes, so each tag value becomes its own attribute set at the metrics backend. The same applies to the default url system tag on dynamic paths, which is why the k6 HTTP documentation recommends setting name to group them.

Service-side metrics have a sharper failure. The OpenTelemetry metrics SDK specification sets a default cardinality limit of 2,000 attribute sets per metric. Past it, for a synchronous instrument with cumulative temporality, measurements for attribute sets not seen before are folded into one data point carrying only otel.metric.overflow=true, while sets seen before the overflow keep being exported. If a service records a user or order ID as a metric attribute during a load test, the ramp can fill the limit, and the steady phase you actually wanted to read then lands in a single anonymous point. The identifiers are not lost to the investigation if you keep them where they cost per record instead of per series: k6 metadata, span attributes or log fields.

Turning baggage into span attributes in the services

The services need nothing new to forward the headers. The SDK configuration specification sets OTEL_PROPAGATORS to tracecontext,baggage by default, so instrumented services extract both headers and inject them into their outgoing calls. What they do not do is record baggage: the OpenTelemetry baggage page says entries are a separate store, not attributes, until you copy them.

For Node.js, the contrib BaggageSpanProcessor copies entries onto every span at start. Its README shows ALLOW_ALL_BAGGAGE_KEYS; pass a predicate instead, so a caller who adds a key cannot add an attribute:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { BaggageSpanProcessor } from '@opentelemetry/baggage-span-processor';

const TEST_CONTEXT = new Set(['test.run', 'journey', 'phase']);

const sdk = new NodeSDK({
  serviceName: 'payment',
  spanProcessors: [
    // ...your exporting span processor first
    new BaggageSpanProcessor((key) => TEST_CONTEXT.has(key)),
  ],
});
sdk.start();

The stub in this article does the equivalent by hand: it parses baggage, keeps the same three keys, and writes them to each span and to each JSON log line along with the trace ID. Logs need that explicit step only where the logging library is not bridged to OpenTelemetry; the context propagation concepts page notes that OpenTelemetry SDKs can inject the trace and span ID into log records automatically.

Two limits apply to what the services do with these values. The baggage page warns that there are no built-in integrity checks, so any caller can send test.run=…; use the keys to label telemetry, never to change behaviour such as skipping a fraud check for "test" traffic. And add journey and phase to service metrics only if you need them there, because their value sets are fixed. test.run gains a new value every run, so on a long-lived metrics backend it adds a set of series per run; spans and logs are the cheaper place for it.

Stopping test context at your network edge

Automatic propagation is also how test context reaches people who should not see it. The OpenTelemetry baggage page says automatic instrumentation includes baggage in most of a service's network requests and that downstream services could propagate it outside your network. The W3C Baggage specification tells application owners to make sure baggage is not present in requests that cross trust boundaries. A payment provider receiving test.run and phase learns that you are load testing and when; a header carrying a user ID would hand it personal data.

The stub's payment service calls a stand-in provider on every charge. Its outgoing-call helper marks that call as egress and drops baggage from it, while the internal calls keep it:

function call(port, path, ctx, bag, { egress = false } = {}) {
  const headers = { traceparent: `00-${ctx.traceId}-${ctx.spanId}-${ctx.sampled ? '01' : '00'}` };
  const baggage = Object.entries(bag).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join(',');
  if (baggage && !(egress && STRIP_AT_EGRESS)) headers.baggage = baggage;
  return new Promise((resolve, reject) => {
    http.request({ host: '127.0.0.1', port, path, method: 'POST', headers }, (r) => {
      r.resume(); r.on('end', () => resolve(r.statusCode));
    }).on('error', reject).end();
  });
}

With stripping turned off, all 1,000 provider calls in a run carried baggage: test.run=perf-2026-10-24-b,journey=checkout,phase=ramp or its steady equivalent. With it on, none of the 1,000 did. In an instrumented service, do the same in whatever owns outbound calls to third parties: a client wrapper, an egress proxy that removes the header, or code that calls the provider under a context with baggage cleared, which the OpenTelemetry JS API exposes as propagation.deleteBaggage(context).

traceparent still reached the provider in both runs. W3C Trace Context forbids putting personal or sensitive data in it and calls the remaining correlation risk theoretical, but it allows a service at a trust boundary to restart the trace with new IDs. The cost of restarting is that the provider's own records can no longer be joined to your trace, so do it only where that join is not needed.

Check the k6 side too. A header such as baggage: user=acct-0042,email=… in the script sends exactly the data both specifications warn about to every service the request touches.

Sampling before the response exists

The library's sampling option sets the sampled flag in traceparent with Math.random() < samplingRate, once per request, before the request is sent. The trace SDK specification makes the default sampler ParentBased(root=AlwaysOn), and a parent-based sampler applies AlwaysOff when the remote parent is not sampled. So when k6 marks a request unsampled, every default-configured service drops its spans, and a tail sampler further down the pipeline never receives them.

The same checkout test, rerun with -e SAMPLING=0.1:

requests=2000 distinct trace ids=2000 all start dc0718: true
failed checkouts=20 with a recorded trace=2
checkout traces recorded=91 of 1000

Twenty checkouts failed; traces exist for two of them. The stub's logs, which are not sampled, still had all 20 failures with their trace IDs, so the evidence existed and the traces did not. Sampling also distorted the span numbers: only 18 payment spans survived from the ramp, and one 929 ms failure among them became the ramp's p95.

The OpenTelemetry sampling page states the limitation directly: with head sampling alone you cannot ensure that all traces with an error are sampled. Leave k6's sampling at its default of 1 and make the decision in the Collector, after the trace is complete. Every policy below uses a type and fields documented in the tail sampling processor's README at v0.163.0:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 200000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-client-errors
        type: numeric_attribute
        numeric_attribute: { key: http.response.status_code, min_value: 400, max_value: 499 }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

A trace is kept when any policy returns a sample decision. The second policy exists because the HTTP semantic conventions leave span status unset for 4xx responses on server spans, so a 409 from checkout would pass the error policy unnoticed. The latency policy measures from the earliest span start to the latest span end in the trace.

This moves the cost to the Collector, and a load test is when it is highest. The processor holds spans in memory until decision_wait expires; the README requires all spans of a trace to reach the same Collector instance, and stores at most num_traces traces in a circular buffer, 50,000 by default. When a new trace arrives at a full buffer, the oldest is removed before any decision is made. Because the k6 library creates one trace per request, 2,000 requests per second with the default 30-second decision_wait means 60,000 open traces, more than the default holds. Size num_traces to at least your peak request rate multiplied by decision_wait, with headroom for the traces your services start themselves, and watch otelcol_processor_tail_sampling_sampling_trace_dropped_too_early during the run. A non-zero value means the Collector dropped traces before sampling them, which can include the failures you needed.

A failing threshold, its trace and its logs

The first run, at full sampling, printed this for the cohort the thresholds select:

http_req_duration{phase:steady,name:checkout}
✓ 'p(95)<300' p(95)=66ms

http_req_failed{phase:steady,name:checkout}
✗ 'rate<0.01' rate=2.00%
...
http_req_failed....................: 1.00%  20 out of 2000
  { phase:steady,name:checkout }...: 2.00%  16 out of 800

Across all 2,000 requests, which include cart calls that never fail in this stub, the failure rate was 1.00%. Restricted to steady-phase checkouts it was 2.00%, sixteen failures in 800 requests; a gate on the unfiltered rate would have judged a diluted number. This is the metric slice. To go from it to a trace, join k6's per-request output to the span and log stores by trace ID:

// join.mjs: node join.mjs run.json spans.jsonl logs.jsonl
// Finds the slowest steady-phase checkout in k6's output and prints its trace and logs.
import { readFileSync } from 'node:fs';

const [k6File, spanFile, logFile] = process.argv.slice(2);
const lines = (f) => readFileSync(f, 'utf8').trim().split('\n').map((l) => JSON.parse(l));

const checkouts = lines(k6File).filter((l) => l.type === 'Point' && l.metric === 'http_req_duration'
  && l.data.tags.name === 'checkout' && l.data.tags.phase === 'steady');
const slowest = checkouts.reduce((a, b) => (b.data.value > a.data.value ? b : a));
const traceId = slowest.data.metadata.trace_id;
console.log(`k6: ${slowest.data.value.toFixed(1)} ms  status=${slowest.data.tags.status}  trace_id=${traceId}`);

const spans = lines(spanFile).filter((s) => s.traceId === traceId);
const byParent = Object.groupBy(spans, (s) => s.parentId ?? 'none');
const ids = new Set(spans.map((s) => s.spanId));
const t0 = Math.min(...spans.map((s) => s.start));
const print = (s, depth) => {
  const bar = ' '.repeat(Math.round((s.start - t0) / 20)) + '#'.repeat(Math.max(1, Math.round((s.end - s.start) / 20)));
  console.log(`${'  '.repeat(depth)}${s.service.padEnd(12 - 2 * depth)} ${String(s.end - s.start).padStart(4)} ms ${s.status}  ${bar}`);
  for (const c of byParent[s.spanId] ?? []) print(c, depth + 1);
};
for (const root of spans.filter((s) => !ids.has(s.parentId))) {
  console.log(`${root.service} parent span ${root.parentId} recorded: ${ids.has(root.parentId)}`);
  print(root, 0);
}
for (const l of lines(logFile).filter((l) => l.trace_id === traceId)) {
  console.log(`log ${l.service} status=${l.status} phase=${l.phase} test.run=${l['test.run']}`);
}

k6: 963.2 ms  status=503  trace_id=dc071818dfea5ca3c16c9422504d92dd
gateway parent span ec75339ddeb8d0b2 recorded: false
gateway       962 ms 503  ################################################
  inventory    22 ms 200  #
    db          6 ms 200   #
  payment     940 ms 503   ###############################################
    provider   37 ms 200   ##
log db status=200 phase=steady test.run=perf-2026-10-24-a
log inventory status=200 phase=steady test.run=perf-2026-10-24-a
log provider status=200 phase=undefined test.run=undefined
log payment status=503 phase=steady test.run=perf-2026-10-24-a
log gateway status=503 phase=steady test.run=perf-2026-10-24-a

Read it top down. k6 measured 963.2 ms and the gateway span 962 ms, so the time was spent in the system, not the load generator. Inventory and its database write took 22 ms. Payment took 940 ms, and the provider it called answered in 37 ms, so roughly 900 ms passed inside payment after the provider replied and before it returned 503. That is where the time moved, and it is a different finding from "the provider is slow" or "the database is slow", both of which a time-window comparison could have suggested. In the stub this is deliberate: every 50th charge stalls for 900 ms and fails. In your system the span tells you which service's code to open.

The log lines carry the run and phase, so a query for test.run=perf-2026-10-24-a and phase=steady returns this cohort and nothing from the synthetic monitor or the other teams. The provider's line has neither, because baggage was stripped before the call; that is the egress rule working. The second line of output is the missing k6 parent from earlier.

An AI assistant can usefully summarise span patterns across hundreds of such traces, for example which service's self-time grew between ramp and steady. The selection that feeds it should not be the assistant's job. The cohort query, the sampling policies and the egress rule are configuration that someone reviews, so the evidence an assistant reads was chosen deterministically, and any cause it proposes should cite trace IDs you can open.

Four symptoms and what each one points to

Traces exist for the run, but not for its failures. Sampling happened before the outcome was known. Check the sampling option in the script, then take the trace IDs your services logged for failed requests and look them up: if the logs have them and the trace store does not, the spans were never recorded.

The first service's spans show a parent that never arrives. That is the k6 library's random parent ID. Nothing is wrong; filter or group on the service's server span instead.

Metric series counts grow with every run, or an otel.metric.overflow point appears. An identifier became a tag or attribute. Find the attribute whose distinct values track request or user count, and move it to metadata, spans or logs.

Tail-sampled traces thin out only at peak load. The Collector buffer is too small for one trace per request. Check sampling_trace_dropped_too_early and raise num_traces, or lower decision_wait if your traces finish quickly.

The API Response Time Analyzer helps you look at the latency distribution of a cohort rather than one percentile, and the HTTP Status Code Reference settles whether a failure code belongs in the error policy or the 4xx one.

Apply this now

Pick the load test that gates your next release. Add test.run from the CI job, journey and phase as k6 tags, and send the same three in a baggage header with the http-instrumentation-tempo library at its default sampling. Add --out json so every request keeps its trace ID. In each service, copy only those three baggage keys onto spans and logs, and confirm by running once against a stand-in for each external dependency that no baggage header reaches it. Set the Collector's tail sampling to keep errors, 4xx, slow traces and a baseline, with num_traces sized from your peak rate. Then make one controlled failure and keep three things together as the run's evidence: the per-phase threshold line, the waterfall of the slowest failing request in that phase, and the log query for the same test.run and phase that returns its lines.

Questions about tracing load tests

Should the script use the Jaeger propagator?

Only if your services understand nothing but uber-trace-id. The OpenTelemetry context propagation page names W3C TraceContext as the default propagator, so w3c matches what instrumented services expect.

Can k6 send its own metrics to the same Collector?

Yes, with --out opentelemetry, which defaults to gRPC on localhost:4317. Trends become histograms, and each rate becomes one counter with a condition attribute of zero or nonzero. All tags become attributes, so the cardinality rule above applies to your k6 tags as much as to service attributes.

Can a threshold reference a trace ID?

No. Thresholds select on tags, and the k6 documentation says metadata cannot be used in thresholds. That is the right split: thresholds judge a cohort, and the trace ID is how you inspect one member of it afterwards.

Can the whole journey be one trace?

Not with this library: both instrumentHTTP and the Client class generate a new trace ID inside every request call. Select a journey through its journey, phase and test.run attributes, and use the trace ID for the single request you want to inspect. The Client class is still useful for a different reason: it lets you trace the requests that matter and send the rest through the plain k6/http module.

Primary references

• k6 HTTP instrumentation for Tempo: the jslib that adds trace context to k6 requests, the W3C and Jaeger propagators, and migration from k6/experimental/tracing.
• Tempo instrumentation options: the propagator and sampling options and their defaults.
• The instrumented HTTP Client class: selective instrumentation and the trace_id added to output metadata.
• Source of http-instrumentation-tempo 1.0.1 at commit ad7d1a4: one trace per request, the dc0718 prefix, the random parent ID, the per-request sampling decision, and http.batch left unpatched.
• Execution context API for k6 scripts: vu.metrics.tags, and metadata as the place for high-cardinality data that thresholds cannot use.
• Tags and groups in k6: system and scenario tags, and tagWithCurrentStageIndex for stage tags.
• Making HTTP requests in k6: grouping dynamic URLs under one name tag.
• k6 OpenTelemetry output: tags mapped to attributes, metric type mapping and exporter defaults.
• k6 v2.3.0 release notes: the version that produced the output in this article.
• W3C Trace Context Level 1 Recommendation: the traceparent fields, the sampled flag, restarting a trace, and the privacy rules for both headers.
• W3C Baggage, Candidate Recommendation Snapshot: the header format, the 64-member and 8,192-byte limits, and keeping baggage out of requests that cross trust boundaries.
• Baggage as an OpenTelemetry signal: baggage is not attributes until copied, automatic propagation to outgoing requests, and no integrity checks.
• OpenTelemetry JS propagation API at api/v1.9.1: propagation.deleteBaggage(context), which returns a context without baggage.
• How OpenTelemetry moves context between services: W3C TraceContext as the default propagator and trace IDs injected into logs.
• OpenTelemetry sampling concepts: head sampling cannot guarantee that traces with errors are kept, and the operating cost of tail sampling.
• Tracing SDK specification, sampling section: the default ParentBased(root=AlwaysOn) sampler and AlwaysOff for unsampled remote parents.
• SDK environment variable specification: OTEL_PROPAGATORS defaulting to tracecontext,baggage.
• Metrics SDK specification, cardinality limits: the default limit of 2,000 and the otel.metric.overflow attribute set.
• Semantic conventions for HTTP spans: span status left unset for 4xx on server spans and set to Error for 5xx.
• Tail sampling processor README at Collector contrib v0.163.0: policy types, decision_wait and num_traces defaults, same-instance routing, and the dropped-too-early metric.
• BaggageSpanProcessor README at release 0.5.0: copying baggage entries to span attributes with a key predicate.