test.run, set by CI), the journey and the phase. Put them on k6 metrics as tags and send them to the services in a W3C baggage header, next to the traceparent that Grafana's http-instrumentation-tempo library adds. In the services, copy only those three keys onto spans and logs, and remove baggage before any call leaves your network. Keep per-request identifiers out of tags and metric attributes. Leave k6's own sampling at 1 and drop traces in the Collector with tail sampling, sized for one trace per request.A checkout load test fails its gate in the steady phase: 2% of payments returned 503. The latency percentile tells you users waited. It does not tell you where the time went, and when you open the tracing backend for that hour you find tens of thousands of checkout traces from the load test, a synthetic monitor and two other teams sharing the environment. Without a way to select the traffic your run sent, the investigation turns into comparing broad time windows and guessing which spans belong to the experiment.
\n\nPart 2 of this series turned objectives into k6 thresholds that fail CI for the right reasons, and Part 4 split the workload into one scenario per journey. This part connects a failing threshold to the spans and log lines behind it, for the same bounded set of requests, without creating a metric series per user or leaking test context to a payment provider.
\n\nhttp-instrumentation-tempo jslib, version 1.0.1, whose source in the grafana/jslib.k6.io repository last changed on 25 April 2025 (commit ad7d1a4). The older k6/experimental/tracing import now stops the test with \"k6/experimental/tracing has been removed\". OpenTelemetry specification v1.61.0, Collector contrib v0.163.0 (where the tail sampling processor is beta), and @opentelemetry/baggage-span-processor 0.5.0 are current. All output below came from k6 v2.3.0 on macOS arm64 driving a Node.js 26.0.0 stub of four services. The stub propagates traceparent and baggage and records a span only when the incoming sampled flag is set, which is what OpenTelemetry's default sampler does. The SDK and Collector configuration shown later was checked against their documentation, not run.One call in the init context replaces the request functions of the k6/http module with traced versions. It does not replace http.batch, so requests sent in a batch carry no trace context:
import tempo from 'https://jslib.k6.io/http-instrumentation-tempo/1.0.1/index.js';\n\ntempo.instrumentHTTP({ propagator: 'w3c' }); // the default; 'jaeger' sends uber-trace-id instead\nThe library's source shows four behaviours worth knowing before you query anything, and the test run confirmed each of them.
\n\nEvery request is its own trace. The library generates a new trace ID inside each request() call. The checkout journey below sends two requests per iteration, and the run's 2,000 requests carried 2,000 distinct trace IDs. A journey is not a trace. You find one request by its trace ID and the whole journey by its context attributes, which is why the next section exists.
The IDs are recognisable. Each trace ID starts with the fixed prefix dc0718, followed by bytes derived from the clock and then random bytes. All 2,000 IDs in the run started that way, which lets you recognise k6-originated traces at a glance.
The parent span never arrives. The parent-id field in traceparent is 16 random hex characters, and k6 exports no span with that ID. The first service's server span is therefore the top of the recorded trace, pointing at a parent nobody reports. That is expected, not a broken pipeline.
The trace ID goes into metadata, not tags. The library sets exec.vu.metrics.metadata.trace_id for the duration of the call. The k6/execution reference describes metadata as similar to tags but usable for high-cardinality data, unavailable to thresholds, and \"likely handled differently by each output\". With --out json, each data point carries it:
{\"metric\":\"http_req_duration\",\"type\":\"Point\",\"data\":{\"time\":\"2026-10-19T16:51:01.285354+05:30\",\"value\":963.226,\"tags\":{\"error_code\":\"1503\",\"expected_response\":\"false\",\"group\":\"\",\"journey\":\"checkout\",\"method\":\"POST\",\"name\":\"checkout\",\"phase\":\"steady\",\"proto\":\"HTTP/1.1\",\"scenario\":\"steady\",\"status\":\"503\",\"test.run\":\"perf-2026-10-24-a\",\"url\":\"checkout\"},\"metadata\":{\"trace_id\":\"dc071818dfea5ca3c16c9422504d92dd\"}}}\nThe CSV output writes it into a trailing metadata column as trace_id=dc0718…. The end-of-test summary aggregates, so it shows no trace IDs at all; keep a per-request output file whenever you intend to follow a slow request into a trace.
The context that selects your traffic needs values you could write down before the test starts:
\n\ntest.run: one value per run, supplied by CI so it matches the job that produced it, for example perf-2026-10-24-a.journey: one of the scenarios from your workload model, such as checkout or browse.phase: ramp, steady or recovery, matching how the test is built.The script below carries all three. Phases are separate scenarios, each with a phase tag, so every metric the scenario emits is labelled without extra code. The run ID comes in through -e and is validated, because a test with a missing or malformed ID produces data nobody can select later:
// checkout.js: k6 run -e TEST_RUN=perf-2026-10-24-a checkout.js\nimport http from 'k6/http';\nimport exec from 'k6/execution';\nimport { check } from 'k6';\nimport tempo from './tempo.js';\n\nconst BASE = __ENV.BASE_URL || 'http://127.0.0.1:3581';\nconst RUN = __ENV.TEST_RUN;\nif (!/^perf-\\d{4}-\\d{2}-\\d{2}-[a-z0-9]+$/.test(RUN || '')) {\n throw new Error('set TEST_RUN, e.g. -e TEST_RUN=perf-2026-10-24-a');\n}\n\ntempo.instrumentHTTP({ propagator: 'w3c', sampling: Number(__ENV.SAMPLING || 1) });\n\nexport const options = {\n tags: { 'test.run': RUN, journey: 'checkout' },\n scenarios: {\n ramp: {\n executor: 'ramping-arrival-rate',\n exec: 'checkout',\n startRate: 0,\n timeUnit: '1s',\n stages: [{ target: 20, duration: '20s' }],\n preAllocatedVUs: 20,\n tags: { phase: 'ramp' },\n },\n steady: {\n executor: 'constant-arrival-rate',\n exec: 'checkout',\n rate: 20,\n timeUnit: '1s',\n startTime: '20s',\n duration: '40s',\n preAllocatedVUs: 30,\n tags: { phase: 'steady' },\n },\n },\n thresholds: {\n 'http_req_duration{phase:steady,name:checkout}': ['p(95)<300'],\n 'http_req_failed{phase:steady,name:checkout}': ['rate<0.01'],\n },\n};\n\nexport function checkout() {\n const t = exec.vu.metrics.tags;\n const headers = {\n 'Content-Type': 'application/json',\n baggage: `test.run=${RUN},journey=checkout,phase=${t.phase}`,\n };\n http.post(`${BASE}/cart/items`, '{\"sku\":\"SKU-1\"}', { headers, tags: { name: 'cart' } });\n const res = http.post(`${BASE}/checkout`, '{}', { headers, tags: { name: 'checkout' } });\n check(res, { 'order placed': (r) => r.status === 201 });\n}\nThe import points at a local copy of the 1.0.1 file from the jslib repository, so the run did not depend on the network; on the day of the check it was byte-for-byte identical to the file served at the jslib.k6.io URL in the first snippet. Two details are easy to get wrong.
\n\nThe baggage header is the W3C format for passing key-value context between services: comma-separated key=value members, with keys restricted to the HTTP token characters, which include the dot. The specification only obliges a platform to propagate all members while there are at most 64 of them and the header stays within 8,192 bytes. Three short keys are far below that, which is one more reason to keep the context small.
The phase comes from exec.vu.metrics.tags, which includes the scenario's own tags. Logged from inside a scenario tagged phase: 'steady', it printed {\"scenario\":\"a\",\"phase\":\"steady\",\"group\":\"\"}. If your phases are stages of a single ramping scenario rather than separate scenarios, the k6 tags documentation shows tagWithCurrentStageIndex from the utils jslib, which adds a numeric stage tag to every sample in the iteration. You then keep a mapping from stage number to phase name next to the script.
The tempting mistake is to tag requests with whatever identifies them: an account, a cart, an order. Here is the same script fragment both ways, run for ten seconds at 20 iterations per second:
\n\n// cardinality.js: the same identifier as a tag, then as metadata\nimport http from 'k6/http';\nimport exec from 'k6/execution';\n\nexport const options = {\n scenarios: {\n c: { executor: 'constant-arrival-rate', rate: 20, timeUnit: '1s', duration: '10s', preAllocatedVUs: 10 },\n },\n};\n\nexport default function () {\n const cart = `cart-${exec.scenario.iterationInTest}`;\n if (__ENV.CART_AS === 'tag') exec.vu.metrics.tags.cart = cart; // wrong\n else exec.vu.metrics.metadata.cart = cart; // right\n http.post('http://127.0.0.1:3581/cart/items', '{}', { tags: { name: 'cart' } });\n}\nCounting distinct tag sets per metric in each JSON output:
\n\nCART_AS=tag\nhttp_req_duration 200\nhttp_reqs 200\niterations 200\nCART_AS=metadata\nhttp_req_duration 1\nhttp_reqs 1\niterations 1\nTwo hundred requests became two hundred series of one sample each. Locally that is only a bigger file. Sent onward, it is a cost: the k6 OpenTelemetry output (--out opentelemetry) maps all k6 tags to OpenTelemetry attributes, so each tag value becomes its own attribute set at the metrics backend. The same applies to the default url system tag on dynamic paths, which is why the k6 HTTP documentation recommends setting name to group them.
Service-side metrics have a sharper failure. The OpenTelemetry metrics SDK specification sets a default cardinality limit of 2,000 attribute sets per metric. Past it, for a synchronous instrument with cumulative temporality, measurements for attribute sets not seen before are folded into one data point carrying only otel.metric.overflow=true, while sets seen before the overflow keep being exported. If a service records a user or order ID as a metric attribute during a load test, the ramp can fill the limit, and the steady phase you actually wanted to read then lands in a single anonymous point. The identifiers are not lost to the investigation if you keep them where they cost per record instead of per series: k6 metadata, span attributes or log fields.
The services need nothing new to forward the headers. The SDK configuration specification sets OTEL_PROPAGATORS to tracecontext,baggage by default, so instrumented services extract both headers and inject them into their outgoing calls. What they do not do is record baggage: the OpenTelemetry baggage page says entries are a separate store, not attributes, until you copy them.
For Node.js, the contrib BaggageSpanProcessor copies entries onto every span at start. Its README shows ALLOW_ALL_BAGGAGE_KEYS; pass a predicate instead, so a caller who adds a key cannot add an attribute:
import { NodeSDK } from '@opentelemetry/sdk-node';\nimport { BaggageSpanProcessor } from '@opentelemetry/baggage-span-processor';\n\nconst TEST_CONTEXT = new Set(['test.run', 'journey', 'phase']);\n\nconst sdk = new NodeSDK({\n serviceName: 'payment',\n spanProcessors: [\n // ...your exporting span processor first\n new BaggageSpanProcessor((key) => TEST_CONTEXT.has(key)),\n ],\n});\nsdk.start();\nThe stub in this article does the equivalent by hand: it parses baggage, keeps the same three keys, and writes them to each span and to each JSON log line along with the trace ID. Logs need that explicit step only where the logging library is not bridged to OpenTelemetry; the context propagation concepts page notes that OpenTelemetry SDKs can inject the trace and span ID into log records automatically.
Two limits apply to what the services do with these values. The baggage page warns that there are no built-in integrity checks, so any caller can send test.run=…; use the keys to label telemetry, never to change behaviour such as skipping a fraud check for \"test\" traffic. And add journey and phase to service metrics only if you need them there, because their value sets are fixed. test.run gains a new value every run, so on a long-lived metrics backend it adds a set of series per run; spans and logs are the cheaper place for it.
Automatic propagation is also how test context reaches people who should not see it. The OpenTelemetry baggage page says automatic instrumentation includes baggage in most of a service's network requests and that downstream services could propagate it outside your network. The W3C Baggage specification tells application owners to make sure baggage is not present in requests that cross trust boundaries. A payment provider receiving test.run and phase learns that you are load testing and when; a header carrying a user ID would hand it personal data.
The stub's payment service calls a stand-in provider on every charge. Its outgoing-call helper marks that call as egress and drops baggage from it, while the internal calls keep it:
function call(port, path, ctx, bag, { egress = false } = {}) {\n const headers = { traceparent: `00-${ctx.traceId}-${ctx.spanId}-${ctx.sampled ? '01' : '00'}` };\n const baggage = Object.entries(bag).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join(',');\n if (baggage && !(egress && STRIP_AT_EGRESS)) headers.baggage = baggage;\n return new Promise((resolve, reject) => {\n http.request({ host: '127.0.0.1', port, path, method: 'POST', headers }, (r) => {\n r.resume(); r.on('end', () => resolve(r.statusCode));\n }).on('error', reject).end();\n });\n}\nWith stripping turned off, all 1,000 provider calls in a run carried baggage: test.run=perf-2026-10-24-b,journey=checkout,phase=ramp or its steady equivalent. With it on, none of the 1,000 did. In an instrumented service, do the same in whatever owns outbound calls to third parties: a client wrapper, an egress proxy that removes the header, or code that calls the provider under a context with baggage cleared, which the OpenTelemetry JS API exposes as propagation.deleteBaggage(context).
traceparent still reached the provider in both runs. W3C Trace Context forbids putting personal or sensitive data in it and calls the remaining correlation risk theoretical, but it allows a service at a trust boundary to restart the trace with new IDs. The cost of restarting is that the provider's own records can no longer be joined to your trace, so do it only where that join is not needed.
Check the k6 side too. A header such as baggage: user=acct-0042,email=… in the script sends exactly the data both specifications warn about to every service the request touches.
The library's sampling option sets the sampled flag in traceparent with Math.random() < samplingRate, once per request, before the request is sent. The trace SDK specification makes the default sampler ParentBased(root=AlwaysOn), and a parent-based sampler applies AlwaysOff when the remote parent is not sampled. So when k6 marks a request unsampled, every default-configured service drops its spans, and a tail sampler further down the pipeline never receives them.
The same checkout test, rerun with -e SAMPLING=0.1:
requests=2000 distinct trace ids=2000 all start dc0718: true\nfailed checkouts=20 with a recorded trace=2\ncheckout traces recorded=91 of 1000\nTwenty checkouts failed; traces exist for two of them. The stub's logs, which are not sampled, still had all 20 failures with their trace IDs, so the evidence existed and the traces did not. Sampling also distorted the span numbers: only 18 payment spans survived from the ramp, and one 929 ms failure among them became the ramp's p95.
\n\nThe OpenTelemetry sampling page states the limitation directly: with head sampling alone you cannot ensure that all traces with an error are sampled. Leave k6's sampling at its default of 1 and make the decision in the Collector, after the trace is complete. Every policy below uses a type and fields documented in the tail sampling processor's README at v0.163.0:
processors:\n tail_sampling:\n decision_wait: 10s\n num_traces: 200000\n policies:\n - name: keep-errors\n type: status_code\n status_code: { status_codes: [ERROR] }\n - name: keep-client-errors\n type: numeric_attribute\n numeric_attribute: { key: http.response.status_code, min_value: 400, max_value: 499 }\n - name: keep-slow\n type: latency\n latency: { threshold_ms: 500 }\n - name: baseline\n type: probabilistic\n probabilistic: { sampling_percentage: 10 }\nA trace is kept when any policy returns a sample decision. The second policy exists because the HTTP semantic conventions leave span status unset for 4xx responses on server spans, so a 409 from checkout would pass the error policy unnoticed. The latency policy measures from the earliest span start to the latest span end in the trace.
\n\nThis moves the cost to the Collector, and a load test is when it is highest. The processor holds spans in memory until decision_wait expires; the README requires all spans of a trace to reach the same Collector instance, and stores at most num_traces traces in a circular buffer, 50,000 by default. When a new trace arrives at a full buffer, the oldest is removed before any decision is made. Because the k6 library creates one trace per request, 2,000 requests per second with the default 30-second decision_wait means 60,000 open traces, more than the default holds. Size num_traces to at least your peak request rate multiplied by decision_wait, with headroom for the traces your services start themselves, and watch otelcol_processor_tail_sampling_sampling_trace_dropped_too_early during the run. A non-zero value means the Collector dropped traces before sampling them, which can include the failures you needed.
The first run, at full sampling, printed this for the cohort the thresholds select:
\n\nhttp_req_duration{phase:steady,name:checkout}\n✓ 'p(95)<300' p(95)=66ms\n\nhttp_req_failed{phase:steady,name:checkout}\n✗ 'rate<0.01' rate=2.00%\n...\nhttp_req_failed....................: 1.00% 20 out of 2000\n { phase:steady,name:checkout }...: 2.00% 16 out of 800\nAcross all 2,000 requests, which include cart calls that never fail in this stub, the failure rate was 1.00%. Restricted to steady-phase checkouts it was 2.00%, sixteen failures in 800 requests; a gate on the unfiltered rate would have judged a diluted number. This is the metric slice. To go from it to a trace, join k6's per-request output to the span and log stores by trace ID:
\n\n// join.mjs: node join.mjs run.json spans.jsonl logs.jsonl\n// Finds the slowest steady-phase checkout in k6's output and prints its trace and logs.\nimport { readFileSync } from 'node:fs';\n\nconst [k6File, spanFile, logFile] = process.argv.slice(2);\nconst lines = (f) => readFileSync(f, 'utf8').trim().split('\\n').map((l) => JSON.parse(l));\n\nconst checkouts = lines(k6File).filter((l) => l.type === 'Point' && l.metric === 'http_req_duration'\n && l.data.tags.name === 'checkout' && l.data.tags.phase === 'steady');\nconst slowest = checkouts.reduce((a, b) => (b.data.value > a.data.value ? b : a));\nconst traceId = slowest.data.metadata.trace_id;\nconsole.log(`k6: ${slowest.data.value.toFixed(1)} ms status=${slowest.data.tags.status} trace_id=${traceId}`);\n\nconst spans = lines(spanFile).filter((s) => s.traceId === traceId);\nconst byParent = Object.groupBy(spans, (s) => s.parentId ?? 'none');\nconst ids = new Set(spans.map((s) => s.spanId));\nconst t0 = Math.min(...spans.map((s) => s.start));\nconst print = (s, depth) => {\n const bar = ' '.repeat(Math.round((s.start - t0) / 20)) + '#'.repeat(Math.max(1, Math.round((s.end - s.start) / 20)));\n console.log(`${' '.repeat(depth)}${s.service.padEnd(12 - 2 * depth)} ${String(s.end - s.start).padStart(4)} ms ${s.status} ${bar}`);\n for (const c of byParent[s.spanId] ?? []) print(c, depth + 1);\n};\nfor (const root of spans.filter((s) => !ids.has(s.parentId))) {\n console.log(`${root.service} parent span ${root.parentId} recorded: ${ids.has(root.parentId)}`);\n print(root, 0);\n}\nfor (const l of lines(logFile).filter((l) => l.trace_id === traceId)) {\n console.log(`log ${l.service} status=${l.status} phase=${l.phase} test.run=${l['test.run']}`);\n}\nk6: 963.2 ms status=503 trace_id=dc071818dfea5ca3c16c9422504d92dd\ngateway parent span ec75339ddeb8d0b2 recorded: false\ngateway 962 ms 503 ################################################\n inventory 22 ms 200 #\n db 6 ms 200 #\n payment 940 ms 503 ###############################################\n provider 37 ms 200 ##\nlog db status=200 phase=steady test.run=perf-2026-10-24-a\nlog inventory status=200 phase=steady test.run=perf-2026-10-24-a\nlog provider status=200 phase=undefined test.run=undefined\nlog payment status=503 phase=steady test.run=perf-2026-10-24-a\nlog gateway status=503 phase=steady test.run=perf-2026-10-24-a\nRead it top down. k6 measured 963.2 ms and the gateway span 962 ms, so the time was spent in the system, not the load generator. Inventory and its database write took 22 ms. Payment took 940 ms, and the provider it called answered in 37 ms, so roughly 900 ms passed inside payment after the provider replied and before it returned 503. That is where the time moved, and it is a different finding from \"the provider is slow\" or \"the database is slow\", both of which a time-window comparison could have suggested. In the stub this is deliberate: every 50th charge stalls for 900 ms and fails. In your system the span tells you which service's code to open.
\n\nThe log lines carry the run and phase, so a query for test.run=perf-2026-10-24-a and phase=steady returns this cohort and nothing from the synthetic monitor or the other teams. The provider's line has neither, because baggage was stripped before the call; that is the egress rule working. The second line of output is the missing k6 parent from earlier.
An AI assistant can usefully summarise span patterns across hundreds of such traces, for example which service's self-time grew between ramp and steady. The selection that feeds it should not be the assistant's job. The cohort query, the sampling policies and the egress rule are configuration that someone reviews, so the evidence an assistant reads was chosen deterministically, and any cause it proposes should cite trace IDs you can open.
\n\nTraces exist for the run, but not for its failures. Sampling happened before the outcome was known. Check the sampling option in the script, then take the trace IDs your services logged for failed requests and look them up: if the logs have them and the trace store does not, the spans were never recorded.
The first service's spans show a parent that never arrives. That is the k6 library's random parent ID. Nothing is wrong; filter or group on the service's server span instead.
\n\nMetric series counts grow with every run, or an otel.metric.overflow point appears. An identifier became a tag or attribute. Find the attribute whose distinct values track request or user count, and move it to metadata, spans or logs.
Tail-sampled traces thin out only at peak load. The Collector buffer is too small for one trace per request. Check sampling_trace_dropped_too_early and raise num_traces, or lower decision_wait if your traces finish quickly.
The API Response Time Analyzer helps you look at the latency distribution of a cohort rather than one percentile, and the HTTP Status Code Reference settles whether a failure code belongs in the error policy or the 4xx one.
\n\nPick the load test that gates your next release. Add test.run from the CI job, journey and phase as k6 tags, and send the same three in a baggage header with the http-instrumentation-tempo library at its default sampling. Add --out json so every request keeps its trace ID. In each service, copy only those three baggage keys onto spans and logs, and confirm by running once against a stand-in for each external dependency that no baggage header reaches it. Set the Collector's tail sampling to keep errors, 4xx, slow traces and a baseline, with num_traces sized from your peak rate. Then make one controlled failure and keep three things together as the run's evidence: the per-phase threshold line, the waterfall of the slowest failing request in that phase, and the log query for the same test.run and phase that returns its lines.
Only if your services understand nothing but uber-trace-id. The OpenTelemetry context propagation page names W3C TraceContext as the default propagator, so w3c matches what instrumented services expect.
Yes, with --out opentelemetry, which defaults to gRPC on localhost:4317. Trends become histograms, and each rate becomes one counter with a condition attribute of zero or nonzero. All tags become attributes, so the cardinality rule above applies to your k6 tags as much as to service attributes.
No. Thresholds select on tags, and the k6 documentation says metadata cannot be used in thresholds. That is the right split: thresholds judge a cohort, and the trace ID is how you inspect one member of it afterwards.
\n\nNot with this library: both instrumentHTTP and the Client class generate a new trace ID inside every request call. Select a journey through its journey, phase and test.run attributes, and use the trace ID for the single request you want to inspect. The Client class is still useful for a different reason: it lets you trace the requests that matter and send the rest through the plain k6/http module.
k6/experimental/tracing.propagator and sampling options and their defaults.trace_id added to output metadata.dc0718 prefix, the random parent ID, the per-request sampling decision, and http.batch left unpatched.vu.metrics.tags, and metadata as the place for high-cardinality data that thresholds cannot use.tagWithCurrentStageIndex for stage tags.name tag.traceparent fields, the sampled flag, restarting a trace, and the privacy rules for both headers.propagation.deleteBaggage(context), which returns a context without baggage.ParentBased(root=AlwaysOn) sampler and AlwaysOff for unsampled remote parents.OTEL_PROPAGATORS defaulting to tracecontext,baggage.otel.metric.overflow attribute set.decision_wait and num_traces defaults, same-instance routing, and the dropped-too-early metric.