A release goes to 5% of traffic. The service dashboard shows an error ratio of 0.025%, a quarter of what a 99.9% objective allows, so the rollout continues. Every customer who applies a coupon at checkout on the new version gets a 500. Over the same minute, the new version's own error ratio was 0.49%, about five times the objective's allowance. Both numbers come from the same 24,003 requests in the lab behind this article: the service view divided 6 failures by all of them, the canary view by the 1,216 the new version served.
\n\nThe earlier parts of this series tested before production, from thresholds that fail CI for the right reasons to capacity. Google's SRE workbook is plain about where that stops: test environments differ from production, and some defects only appear under real traffic. Canary testing needs the same discipline as a k6 gate: an objective, a control, enough samples and a rule that fails when it should.
\n\nThe dilution is plain arithmetic. A canary serving a share w of traffic with error ratio e adds w × e to the service's error ratio. The SRE workbook's own example: a release failing 20% of requests, canaried to 5%, shows up as a 1% service-wide error rate. The chapter then notes how hard that is to see against the overall trend.
\n\nThe same arithmetic tells you that your normal SLO alerts are blind during a canary. The workbook's recommended fast-burn page fires at a burn rate of 14.4, which for a 99.9% objective is a 1.44% error ratio. A 1% canary that fails every request moves the service error ratio by 1%, a burn rate of 10, so on its own it can never trigger that page. At 5%, the canary would have to fail 28.8% of its requests (0.0144 ÷ 0.05) before the page fired. In the coupon run, the canary's 0.49% added up to a service-wide burn rate of 0.25.
\n\nHere is stage 2 of that run, with canary and baseline at 5% each, split by the version that served each request:
\n\nwindow 92-152s\n all n= 24003 5xx= 6 (0.02%) order_p95=43.8ms cpu/req=0.216ms\n stable n= 21604 5xx= 0 (0.00%) order_p95=43.8ms cpu/req=0.215ms\n baseline n= 1183 5xx= 0 (0.00%) order_p95=43.8ms cpu/req=0.216ms\n canary n= 1216 5xx= 6 (0.49%) order_p95=43.9ms cpu/req=0.217msLatency and CPU did not move, because the defect only touched one rare path: 20% of requests are orders and 3% of orders carry a coupon, so it hit about 0.6% of the canary's traffic. Whatever you compare, it has to be split by version. If the service already exports OpenTelemetry, the service.version resource attribute is the standard place for that label, and part 5 on connecting load tests to traces covers carrying test context into traces and logs without exploding its cardinality.
A canary verdict is a comparison, and it is only as good as what the canary is compared with.
\n\nBefore and after. Compare the service after the deploy with the service before it. The workbook warns against it, because time is one of the biggest sources of change in observed metrics. To show why, the lab added 25 ms to every version, old and new, from second 31, the moment the rollout began, standing in for a batch job, a traffic shift or a slower dependency. Order p95 across the service went from 44.0 ms before the rollout to 69.0 ms after it. A before-and-after gate at 1.25× would have rolled back a release with nothing wrong in it. Over stage 2, canary against baseline read 69.3 ms against 68.9 ms, and the gate promoted all three stages.
\n\nAgainst the stable fleet. This is better, because both sides share the clock. It still compares pods that started minutes ago with pods that have run for days. Spinnaker's canary best-practices page asks for a baseline deployed at the same time, at the same size and with the same traffic, and names cache warm-up time and heap size among the factors this removes.
\n\nAgainst a fresh baseline. Start new pods of the current version alongside the canary, and give both the same traffic weight. Argo Rollouts builds this in as an Experiment step, and the lab's gate does the same on every stage.
\n\nThe window matters as much as the cohort. In another run, the stable fleet threw 503s on 2.70% of requests for the first 25 seconds, before any canary existed, and the coupon defect then went out as the canary. Over stage 2 the canary's error ratio was 0.79% and the baseline's was 0.00%, a delta of 0.79 points. Compared with the stable fleet over a lookback that began at second 0, the control read 0.48% and the delta fell to 0.31 points. The incident had nothing to do with the release, yet it hid 61% of the signal. That is the workbook's rule: metric intervals the same as or shorter than the canary.
\n\nThe workbook's model of a canary's cost is the traffic share multiplied by the time a defect is exposed, and it charges that against the error budget. For a 99.9% objective over 30 days, the budget is 0.1% of requests: the equivalent of 43.2 minutes of total outage. This script prices three typical stages:
\n\n// budget.mjs: share of a 30-day error budget a canary stage can spend.\nconst SLO = 0.999;\nconst PERIOD_MIN = 30 * 24 * 60; // 43,200 minutes\nconst budgetMin = (1 - SLO) * PERIOD_MIN; // 43.2 minutes of total outage\n\nfunction stageCost({ weight, minutes, canaryErrorRate }) {\n const failedMinutes = (weight / 100) * canaryErrorRate * minutes;\n return (failedMinutes / budgetMin) * 100; // percent of the 30-day budget\n}budget: 43.2 minutes of full outage per 30 days\n1% for 30 min: all requests fail, whole stage 0.69% | stopped after 10 min 0.23% | 0.6% failing, whole stage 0.004%\n5% for 30 min: all requests fail, whole stage 3.47% | stopped after 10 min 1.16% | 0.6% failing, whole stage 0.021%\n25% for 60 min: all requests fail, whole stage 34.72% | stopped after 10 min 5.79% | 0.6% failing, whole stage 0.208%Two things follow. First, a stage's real cost is the share multiplied by the time to stop, not by the stage's length. An hour at 25% costs a third of the month's budget only if nothing stops it, so the detection loop matters more than the stage plan. Second, the coupon defect costs almost nothing at any stage; the rules exist to keep a badly broken release away from 25%.
\n\nStages also decide what you can measure. The workbook recommends judging the first, smallest stage on the clearest signals, such as crashes and request failures, and adding metrics that need more data as the population grows. In the lab, stage 1 at 1% gave the canary 228 requests per minute, too few to tell 0.49% from zero. The lab's stage plan follows that order:
\n\n// canary-gate.mjs (excerpt): stages, and two of the five rules\nconst SLO = 0.999; // 99.9% of requests succeed, 30-day window\nconst BUDGET = 1 - SLO; // 0.1% of requests may fail\n\nconst STAGES = [\n { weight: 1, holdS: 60, rules: ['hard', 'control'] },\n { weight: 5, holdS: 60, rules: ['hard', 'control', 'errors', 'latency'] },\n { weight: 25, holdS: 60, rules: ['hard', 'control', 'errors', 'latency', 'cpu'] },\n];\n\nconst RULES = {\n // ...hard: canary 5xx ratio above 20x budget with at least 3 failures\n errors: ({ canary, baseline }) => {\n if (canary.n < 1000 || baseline.n < 1000) return { verdict: 'wait', why: `need 1000 requests per cohort, have ${canary.n}/${baseline.n}` };\n const delta = canary.errors / canary.n - baseline.errors / baseline.n;\n return delta > 2 * BUDGET\n ? { verdict: 'fail', why: `error delta ${(delta * 100).toFixed(2)} pts > ${(2 * BUDGET * 100).toFixed(1)} pts` }\n : { verdict: 'pass', why: `error delta ${(delta * 100).toFixed(2)} pts` };\n },\n control: ({ baseline, stable }) => {\n for (const [name, g] of Object.entries({ baseline, stable })) {\n const ratio = g.errors / g.n;\n if (ratio > 14.4 * BUDGET) return { verdict: 'fail', why: `${name} 5xx ${(ratio * 100).toFixed(2)}% > ${(14.4 * BUDGET * 100).toFixed(2)}% (14.4x burn) during canary` };\n }\n return { verdict: 'pass', why: 'control inside 14.4x burn' };\n },\n // ...latency: order p95 ratio above 1.25 with 200 orders per cohort; cpu: CPU per request ratio above 1.3\n};Every 10 seconds the gate re-reads the request log, evaluates the stage's rules over the canary and baseline cohorts in the same window, and aborts on the first failure. A rule that still lacks its sample floor at the end of a stage returns wait, and the stage pauses for a human instead of promoting. Three releases went through it:
| Release | Stage 1 (1%) | Stage 2 (5%) | Stage 3 (25%) |
|---|---|---|---|
| Coupon orders return 500 | Promoted, 0 of 228 failed | Aborted: error delta 0.49 points, 6 of 1,216 failed | Never reached |
| Extra CPU work on every request | Promoted | Promoted, order p95 ×1.02 | Aborted: 1.249 against 0.216 ms per request (×5.77) |
| No defect | Promoted | Promoted, delta 0.00 points | Promoted, CPU ×0.99 |
The CPU regression did not hurt users, since latency stayed within 2%, which is why the workbook, for its example service, prefers return codes and latency to CPU usage. It still matters at full rollout: an extra millisecond on each of 400 requests a second is about 0.41 of a core, the headroom capacity testing exists to measure. Measure CPU spent serving the request, as the lab does, not host CPU, which the workbook notes also moves with other processes such as log rotation. One gap in the lab's rule: cpu has no sample floor, so it aborted after 958 canary requests. Give it one before relying on it.
Each decision is written down as it is made:
\n\n{\"at_s\":92,\"from_ms\":32002,\"to_ms\":92012,\"stage\":1,\"weight\":1,\"decision\":\"promote\",\"reasons\":[\"hard pass: canary 5xx 0/228\",\"control pass: control inside 14.4x burn\"]}\n{\"at_s\":152,\"from_ms\":92012,\"to_ms\":152029,\"stage\":2,\"weight\":5,\"decision\":\"abort\",\"reasons\":[\"hard pass: canary 5xx 6/1216\",\"control pass: control inside 14.4x burn\",\"errors fail: error delta 0.49 pts > 0.2 pts\",\"latency pass: order p95 x1.00\"]}That file is the evidence a release review needs: the window compared, the rule that fired, and on how many requests. An AI assistant can summarise these records into a readable account of what changed. It should not decide to promote; the rules, thresholds and rollback policy are code a human reviewed before the rollout began.
\n\nThe workbook warns that canary and control may share backends, data stores and other infrastructure, so a bad canary can damage the control and confuse the comparison. The lab reproduced that with one more release. Stage 1 is the first window below, stage 2 the second. The new version wrote product entries to a shared cache in a new shape. It could read both shapes, but the old version could not read the new one:
\n\nwindow 32-92s\n all n= 24002 5xx= 142 (0.59%) order_p95=43.9ms cpu/req=0.219ms\n stable n= 23511 5xx= 141 (0.60%) order_p95=43.9ms cpu/req=0.219ms\n canary n= 242 5xx= 0 (0.00%) order_p95=43.0ms cpu/req=0.220ms\n baseline n= 249 5xx= 1 (0.40%) order_p95=41.3ms cpu/req=0.218ms\nwindow 92-102s\n all n= 4001 5xx= 90 (2.25%) order_p95=44.0ms cpu/req=0.221ms\n stable n= 3581 5xx= 86 (2.40%) order_p95=44.0ms cpu/req=0.221ms\n baseline n= 212 5xx= 4 (1.89%) order_p95=44.8ms cpu/req=0.219ms\n canary n= 208 5xx= 0 (0.00%) order_p95=44.0ms cpu/req=0.224msAll 260 failures in the run were served by the old version. A comparison-only gate reads this as a canary healthier than its control. The 1% canary was also costing more than its own traffic share: 0.59% of all requests failed in stage 1. The gate's control rule, an absolute SLO check on the baseline and the stable fleet, aborted the rollout ten seconds into stage 2. It did not fire in stage 1, though, because 0.60% sits below the 1.44% fast-burn line. A lower threshold, such as 5× (0.5%), would have stopped it by the end of stage 1. The workbook's advice is to keep absolute measures such as the SLO alongside the comparison for this reason. The release itself needed two steps: first a version that reads both shapes but still writes the old one, then the new writer once no old reader remains.
Argo Rollouts' Experiment step runs a baseline and a canary side by side, and with a traffic router it can give each a weight. The documented example sends 5% to each and leaves 90% on the old stack. Weighted experiments need the SMI, ALB or Istio traffic router. Without a router, Argo approximates weights by replica count, choosing the whole number of pods with the smallest difference and rounding up on a tie. By that rule, a 5% step on 10 replicas runs 1 canary pod, which is 10% of the pods.
\n\n# Rollout (excerpt): stage 2 as a weighted experiment\nstrategy:\n canary:\n trafficRouting:\n istio:\n virtualService:\n name: shop-vsvc\n steps:\n - experiment:\n duration: 30m\n templates:\n - name: baseline\n specRef: stable\n weight: 5\n - name: canary\n specRef: canary\n weight: 5\n analyses:\n - name: errors\n templateName: canary-vs-baseline-errors\n args:\n - name: baseline-hash\n value: \"{{templates.baseline.podTemplateHash}}\"\n - name: canary-hash\n value: \"{{templates.canary.podTemplateHash}}\"The experiment's pods get pod-template hashes of their own, different from the Rollout's stable and canary ReplicaSets, which is what lets the query below select them. Argo's Kayenta page passes podTemplateHashValue: Stable and Latest instead, which selects the Rollout's ReplicaSets rather than the experiment's baseline. Copy the Experiment page's variables if you want the fresh baseline.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: canary-vs-baseline-errors\nspec:\n args:\n - name: baseline-hash\n - name: canary-hash\n metrics:\n - name: error-delta\n initialDelay: 5m\n interval: 5m\n count: 5\n # An empty result matches neither condition: Inconclusive, so the rollout pauses.\n successCondition: len(result) > 0 && result[0] <= 0.002\n failureCondition: len(result) > 0 && result[0] > 0.002\n provider:\n prometheus:\n address: http://prometheus.monitoring:9090\n # Assumes the scrape config copies the pod label rollouts-pod-template-hash\n # onto each series as rollouts_pod_template_hash.\n query: |\n (\n (sum(rate(http_requests_total{rollouts_pod_template_hash=\"{{args.canary-hash}}\",code=~\"5..\"}[5m])) or vector(0))\n / sum(rate(http_requests_total{rollouts_pod_template_hash=\"{{args.canary-hash}}\"}[5m]))\n )\n -\n (\n (sum(rate(http_requests_total{rollouts_pod_template_hash=\"{{args.baseline-hash}}\",code=~\"5..\"}[5m])) or vector(0))\n / sum(rate(http_requests_total{rollouts_pod_template_hash=\"{{args.baseline-hash}}\"}[5m]))\n )An AnalysisRun ends Successful, Failed or Inconclusive, which continue, abort or pause the rollout. The or vector(0) stops a canary with no 5xx series from emptying the ratio, while a cohort with no traffic still returns nothing, which matches neither condition and lands on Inconclusive instead of a pass. In the v1.10.1 source, inconclusiveLimit defaults to 0, so one such measurement pauses the rollout. The 5-minute rate window matches the interval, so no measurement reaches far behind the experiment. Read the analysis page's examples before copying them: its error-rate template uses successCondition: result[0] <= 0.95, which passes until more than 95% of requests fail.
Flagger's built-in checks, request success rate and duration, compare the canary's own metrics with a fixed thresholdRange, not with the primary; a delta needs a custom MetricTemplate. Stages are stepWeight up to maxWeight, or a list such as stepWeights: [1, 2, 10, 80]. Flagger gives rollback time as interval × threshold, so its example of 1 minute and 10 leaves a failing canary serving for ten minutes, the \"stopped after\" column in the budget output.
Kayenta, Spinnaker's judge, does the statistics. In the 2026.1.3 source it runs a Mann-Whitney U test at 98% confidence. A metric must be statistically significant before its effect size is even checked, so the judge page notes that a 20% increase can still pass when the confidence interval is wide. Spinnaker's guidance asks for at least 50 data points per metric, and starting points of 3-hour canaries with a marginal score of 75 and a pass score of 95. One detail in the scorer: a metric group with no data at all scores 100, but if half or more of the metrics return no data, the whole canary fails.
\n\nThe coupon path carried about 0.6% of the canary's traffic, so stage 2 saw 6 failures in a minute. The workbook does not count synthetic load as canarying, but it does suggest folding black-box checks into canary analysis. A k6 script aimed only at the canary pods, through the Rollout's canaryService, can push rare paths hard and quickly. Argo Rollouts treats a Kubernetes Job as a metric: exit code 0 passes, anything else fails.
// canary-probe.js: a black-box check sent only to the canary pods (an Argo Rollouts Job metric).\nimport http from 'k6/http';\nimport { check } from 'k6';\n\nconst TARGET = __ENV.CANARY_URL || 'http://127.0.0.1:3481';\n\nexport const options = {\n scenarios: {\n probe: {\n executor: 'constant-arrival-rate',\n rate: 20,\n timeUnit: '1s',\n duration: '1m',\n preAllocatedVUs: 10,\n },\n },\n thresholds: {\n 'http_req_failed{step:coupon_order}': ['rate<0.01'],\n 'http_reqs{step:coupon_order}': ['count>=100'],\n 'http_req_duration{step:product}': ['p(95)<100'],\n },\n};\n\nexport default function () {\n http.get(`${TARGET}/products/${Math.floor(Math.random() * 500)}`, { tags: { step: 'product' } });\n // Exercise the path the production mix reaches rarely: every probe iteration sends a coupon order.\n const res = http.post(`${TARGET}/orders`, JSON.stringify({ sku: 'SKU-1', qty: 1, coupon: 'PROBE-ONLY' }), {\n headers: { 'Content-Type': 'application/json' },\n tags: { step: 'coupon_order' },\n });\n check(res, { 'order placed': (r) => r.status === 201 }, { step: 'coupon_order' });\n}Against the defective canary it sent 1,201 coupon orders in a minute and exited 99:
\n\n http_req_failed{step:coupon_order}\n ✗ 'rate<0.01' rate=100.00%\n\n http_reqs{step:coupon_order}\n ✓ 'count>=100' count=1201\n\nlevel=error msg=\"thresholds on metrics 'http_req_failed{step:coupon_order}' have been crossed\"Against the healthy version it exited 0. The count floor is part 2's safeguard against a tag that receives no samples, whose thresholds pass. The probe has two costs. The Job metric fails on any non-zero exit, so an exception in the script (exit 107) aborts a good release exactly as a real failure (exit 99) does; failing closed is defensible because rollback is cheap, but choose it deliberately. And the probe writes real orders. The workbook warns that artificial load on billing paths has real side effects, so use a test tenant the business has agreed to, or probe read paths only.
Check today whether your metrics can be split by version. Take one SLI with a version label, error ratio is the easiest, and write the comparison as canary against baseline in the same window, with a sample floor per cohort. Price your stage plan with the budget formula, using your real time to stop. Add an absolute SLO check on the control. Then send a deliberately broken build and a healthy one through staging under the same rules, and keep both decision records: a gate that has never aborted is untested. For which statuses count as failures, the HTTP status code reference helps; the workbook suggests leaving 4xx out of canary error counts and catching broken URLs with black-box probes.
\n\nLengthen the stage rather than lowering the floor, and use the probe for the paths you care about. The workbook ties the size and duration a canary needs to request rate and how varied requests are: the more uniform the traffic, the less you need.
\n\nThe workbook is direct: \"We strongly advise running only one canary deployment at a time.\" Overlapping canaries contaminate each other's signals.
\n\nOff-peak lowers the cost of a bad release and its chance of being caught. The workbook notes that performance defects typically appear only under heavy load, so an off-peak canary can pass a release that fails at peak.
\n\nsetWeight and pause steps, replica-count rounding without traffic routing, and canaryService.successCondition and failureCondition, empty-result and NaN handling, Inconclusive runs, initialDelay, and the published error-rate example.specRef, weighted experiment steps for SMI, ALB and Istio, and separate pod-template hashes.inconclusiveLimit defaulting to 0.stepWeight, maxWeight, stepWeights, and rollback time as interval × threshold.thresholdRange, and custom MetricTemplates.or combines vectors, used to keep an absent error series from emptying the ratio.service.version as the version string of the service.