Use Canary Analysis as a Production Test

A release goes to 5% of traffic. The service dashboard shows an error ratio of 0.025%, a quarter of what a 99.9% objective allows, so the rollout continues. Every customer who applies a coupon at checkout on the new version gets a 500. Over the same minute, the new version's own error ratio was 0.49%, about five times the objective's allowance. Both numbers come from the same 24,003 requests in the lab behind this article: the service view divided 6 failures by all of them, the canary view by the 1,216 the new version served.

The earlier parts of this series tested before production, from thresholds that fail CI for the right reasons to capacity. Google's SRE workbook is plain about where that stops: test environments differ from production, and some defects only appear under real traffic. Canary testing needs the same discipline as a k6 gate: an objective, a control, enough samples and a rule that fails when it should.

Why the service view cannot see a small canary

The dilution is plain arithmetic. A canary serving a share w of traffic with error ratio e adds w × e to the service's error ratio. The SRE workbook's own example: a release failing 20% of requests, canaried to 5%, shows up as a 1% service-wide error rate. The chapter then notes how hard that is to see against the overall trend.

The same arithmetic tells you that your normal SLO alerts are blind during a canary. The workbook's recommended fast-burn page fires at a burn rate of 14.4, which for a 99.9% objective is a 1.44% error ratio. A 1% canary that fails every request moves the service error ratio by 1%, a burn rate of 10, so on its own it can never trigger that page. At 5%, the canary would have to fail 28.8% of its requests (0.0144 ÷ 0.05) before the page fired. In the coupon run, the canary's 0.49% added up to a service-wide burn rate of 0.25.

Here is stage 2 of that run, with canary and baseline at 5% each, split by the version that served each request:

window 92-152s
  all      n= 24003  5xx=   6 (0.02%)  order_p95=43.8ms  cpu/req=0.216ms
  stable   n= 21604  5xx=   0 (0.00%)  order_p95=43.8ms  cpu/req=0.215ms
  baseline n=  1183  5xx=   0 (0.00%)  order_p95=43.8ms  cpu/req=0.216ms
  canary   n=  1216  5xx=   6 (0.49%)  order_p95=43.9ms  cpu/req=0.217ms

Latency and CPU did not move, because the defect only touched one rare path: 20% of requests are orders and 3% of orders carry a coupon, so it hit about 0.6% of the canary's traffic. Whatever you compare, it has to be split by version. If the service already exports OpenTelemetry, the service.version resource attribute is the standard place for that label, and part 5 on connecting load tests to traces covers carrying test context into traces and logs without exploding its cardinality.

Choose the control before you choose the metrics

A canary verdict is a comparison, and it is only as good as what the canary is compared with.

Before and after. Compare the service after the deploy with the service before it. The workbook warns against it, because time is one of the biggest sources of change in observed metrics. To show why, the lab added 25 ms to every version, old and new, from second 31, the moment the rollout began, standing in for a batch job, a traffic shift or a slower dependency. Order p95 across the service went from 44.0 ms before the rollout to 69.0 ms after it. A before-and-after gate at 1.25× would have rolled back a release with nothing wrong in it. Over stage 2, canary against baseline read 69.3 ms against 68.9 ms, and the gate promoted all three stages.

Against the stable fleet. This is better, because both sides share the clock. It still compares pods that started minutes ago with pods that have run for days. Spinnaker's canary best-practices page asks for a baseline deployed at the same time, at the same size and with the same traffic, and names cache warm-up time and heap size among the factors this removes.

Against a fresh baseline. Start new pods of the current version alongside the canary, and give both the same traffic weight. Argo Rollouts builds this in as an Experiment step, and the lab's gate does the same on every stage.

The window matters as much as the cohort. In another run, the stable fleet threw 503s on 2.70% of requests for the first 25 seconds, before any canary existed, and the coupon defect then went out as the canary. Over stage 2 the canary's error ratio was 0.79% and the baseline's was 0.00%, a delta of 0.79 points. Compared with the stable fleet over a lookback that began at second 0, the control read 0.48% and the delta fell to 0.31 points. The incident had nothing to do with the release, yet it hid 61% of the signal. That is the workbook's rule: metric intervals the same as or shorter than the canary.

Stage the exposure and price each stage in budget

The workbook's model of a canary's cost is the traffic share multiplied by the time a defect is exposed, and it charges that against the error budget. For a 99.9% objective over 30 days, the budget is 0.1% of requests: the equivalent of 43.2 minutes of total outage. This script prices three typical stages:

// budget.mjs: share of a 30-day error budget a canary stage can spend.
const SLO = 0.999;
const PERIOD_MIN = 30 * 24 * 60;              // 43,200 minutes
const budgetMin = (1 - SLO) * PERIOD_MIN;      // 43.2 minutes of total outage

function stageCost({ weight, minutes, canaryErrorRate }) {
  const failedMinutes = (weight / 100) * canaryErrorRate * minutes;
  return (failedMinutes / budgetMin) * 100;   // percent of the 30-day budget
}

budget: 43.2 minutes of full outage per 30 days
1%  for 30 min: all requests fail, whole stage 0.69% | stopped after 10 min 0.23% | 0.6% failing, whole stage 0.004%
5%  for 30 min: all requests fail, whole stage 3.47% | stopped after 10 min 1.16% | 0.6% failing, whole stage 0.021%
25% for 60 min: all requests fail, whole stage 34.72% | stopped after 10 min 5.79% | 0.6% failing, whole stage 0.208%

Two things follow. First, a stage's real cost is the share multiplied by the time to stop, not by the stage's length. An hour at 25% costs a third of the month's budget only if nothing stops it, so the detection loop matters more than the stage plan. Second, the coupon defect costs almost nothing at any stage; the rules exist to keep a badly broken release away from 25%.

Stages also decide what you can measure. The workbook recommends judging the first, smallest stage on the clearest signals, such as crashes and request failures, and adding metrics that need more data as the population grows. In the lab, stage 1 at 1% gave the canary 228 requests per minute, too few to tell 0.49% from zero. The lab's stage plan follows that order:

// canary-gate.mjs (excerpt): stages, and two of the five rules
const SLO = 0.999;            // 99.9% of requests succeed, 30-day window
const BUDGET = 1 - SLO;       // 0.1% of requests may fail

const STAGES = [
  { weight: 1,  holdS: 60, rules: ['hard', 'control'] },
  { weight: 5,  holdS: 60, rules: ['hard', 'control', 'errors', 'latency'] },
  { weight: 25, holdS: 60, rules: ['hard', 'control', 'errors', 'latency', 'cpu'] },
];

const RULES = {
  // ...hard: canary 5xx ratio above 20x budget with at least 3 failures
  errors: ({ canary, baseline }) => {
    if (canary.n < 1000 || baseline.n < 1000) return { verdict: 'wait', why: `need 1000 requests per cohort, have ${canary.n}/${baseline.n}` };
    const delta = canary.errors / canary.n - baseline.errors / baseline.n;
    return delta > 2 * BUDGET
      ? { verdict: 'fail', why: `error delta ${(delta * 100).toFixed(2)} pts > ${(2 * BUDGET * 100).toFixed(1)} pts` }
      : { verdict: 'pass', why: `error delta ${(delta * 100).toFixed(2)} pts` };
  },
  control: ({ baseline, stable }) => {
    for (const [name, g] of Object.entries({ baseline, stable })) {
      const ratio = g.errors / g.n;
      if (ratio > 14.4 * BUDGET) return { verdict: 'fail', why: `${name} 5xx ${(ratio * 100).toFixed(2)}% > ${(14.4 * BUDGET * 100).toFixed(2)}% (14.4x burn) during canary` };
    }
    return { verdict: 'pass', why: 'control inside 14.4x burn' };
  },
  // ...latency: order p95 ratio above 1.25 with 200 orders per cohort; cpu: CPU per request ratio above 1.3
};

Every 10 seconds the gate re-reads the request log, evaluates the stage's rules over the canary and baseline cohorts in the same window, and aborts on the first failure. A rule that still lacks its sample floor at the end of a stage returns wait, and the stage pauses for a human instead of promoting. Three releases went through it:

The CPU regression did not hurt users, since latency stayed within 2%, which is why the workbook, for its example service, prefers return codes and latency to CPU usage. It still matters at full rollout: an extra millisecond on each of 400 requests a second is about 0.41 of a core, the headroom capacity testing exists to measure. Measure CPU spent serving the request, as the lab does, not host CPU, which the workbook notes also moves with other processes such as log rotation. One gap in the lab's rule: cpu has no sample floor, so it aborted after 958 canary requests. Give it one before relying on it.

Each decision is written down as it is made:

{"at_s":92,"from_ms":32002,"to_ms":92012,"stage":1,"weight":1,"decision":"promote","reasons":["hard pass: canary 5xx 0/228","control pass: control inside 14.4x burn"]}
{"at_s":152,"from_ms":92012,"to_ms":152029,"stage":2,"weight":5,"decision":"abort","reasons":["hard pass: canary 5xx 6/1216","control pass: control inside 14.4x burn","errors fail: error delta 0.49 pts > 0.2 pts","latency pass: order p95 x1.00"]}

That file is the evidence a release review needs: the window compared, the rule that fired, and on how many requests. An AI assistant can summarise these records into a readable account of what changed. It should not decide to promote; the rules, thresholds and rollback policy are code a human reviewed before the rollout began.

When the canary breaks the control

The workbook warns that canary and control may share backends, data stores and other infrastructure, so a bad canary can damage the control and confuse the comparison. The lab reproduced that with one more release. Stage 1 is the first window below, stage 2 the second. The new version wrote product entries to a shared cache in a new shape. It could read both shapes, but the old version could not read the new one:

window 32-92s
  all      n= 24002  5xx= 142 (0.59%)  order_p95=43.9ms  cpu/req=0.219ms
  stable   n= 23511  5xx= 141 (0.60%)  order_p95=43.9ms  cpu/req=0.219ms
  canary   n=   242  5xx=   0 (0.00%)  order_p95=43.0ms  cpu/req=0.220ms
  baseline n=   249  5xx=   1 (0.40%)  order_p95=41.3ms  cpu/req=0.218ms
window 92-102s
  all      n=  4001  5xx=  90 (2.25%)  order_p95=44.0ms  cpu/req=0.221ms
  stable   n=  3581  5xx=  86 (2.40%)  order_p95=44.0ms  cpu/req=0.221ms
  baseline n=   212  5xx=   4 (1.89%)  order_p95=44.8ms  cpu/req=0.219ms
  canary   n=   208  5xx=   0 (0.00%)  order_p95=44.0ms  cpu/req=0.224ms

All 260 failures in the run were served by the old version. A comparison-only gate reads this as a canary healthier than its control. The 1% canary was also costing more than its own traffic share: 0.59% of all requests failed in stage 1. The gate's control rule, an absolute SLO check on the baseline and the stable fleet, aborted the rollout ten seconds into stage 2. It did not fire in stage 1, though, because 0.60% sits below the 1.44% fast-burn line. A lower threshold, such as 5× (0.5%), would have stopped it by the end of stage 1. The workbook's advice is to keep absolute measures such as the SLO alongside the comparison for this reason. The release itself needed two steps: first a version that reads both shapes but still writes the old one, then the new writer once no old reader remains.

Wiring the comparison into a rollout controller

Argo Rollouts' Experiment step runs a baseline and a canary side by side, and with a traffic router it can give each a weight. The documented example sends 5% to each and leaves 90% on the old stack. Weighted experiments need the SMI, ALB or Istio traffic router. Without a router, Argo approximates weights by replica count, choosing the whole number of pods with the smallest difference and rounding up on a tie. By that rule, a 5% step on 10 replicas runs 1 canary pod, which is 10% of the pods.

# Rollout (excerpt): stage 2 as a weighted experiment
strategy:
  canary:
    trafficRouting:
      istio:
        virtualService:
          name: shop-vsvc
    steps:
    - experiment:
        duration: 30m
        templates:
        - name: baseline
          specRef: stable
          weight: 5
        - name: canary
          specRef: canary
          weight: 5
        analyses:
        - name: errors
          templateName: canary-vs-baseline-errors
          args:
          - name: baseline-hash
            value: "{{templates.baseline.podTemplateHash}}"
          - name: canary-hash
            value: "{{templates.canary.podTemplateHash}}"

The experiment's pods get pod-template hashes of their own, different from the Rollout's stable and canary ReplicaSets, which is what lets the query below select them. Argo's Kayenta page passes podTemplateHashValue: Stable and Latest instead, which selects the Rollout's ReplicaSets rather than the experiment's baseline. Copy the Experiment page's variables if you want the fresh baseline.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-vs-baseline-errors
spec:
  args:
  - name: baseline-hash
  - name: canary-hash
  metrics:
  - name: error-delta
    initialDelay: 5m
    interval: 5m
    count: 5
    # An empty result matches neither condition: Inconclusive, so the rollout pauses.
    successCondition: len(result) > 0 && result[0] <= 0.002
    failureCondition: len(result) > 0 && result[0] > 0.002
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        # Assumes the scrape config copies the pod label rollouts-pod-template-hash
        # onto each series as rollouts_pod_template_hash.
        query: |
          (
            (sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[5m])) or vector(0))
            / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m]))
          )
          -
          (
            (sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.baseline-hash}}",code=~"5.."}[5m])) or vector(0))
            / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.baseline-hash}}"}[5m]))
          )

An AnalysisRun ends Successful, Failed or Inconclusive, which continue, abort or pause the rollout. The or vector(0) stops a canary with no 5xx series from emptying the ratio, while a cohort with no traffic still returns nothing, which matches neither condition and lands on Inconclusive instead of a pass. In the v1.10.1 source, inconclusiveLimit defaults to 0, so one such measurement pauses the rollout. The 5-minute rate window matches the interval, so no measurement reaches far behind the experiment. Read the analysis page's examples before copying them: its error-rate template uses successCondition: result[0] <= 0.95, which passes until more than 95% of requests fail.

Flagger's built-in checks, request success rate and duration, compare the canary's own metrics with a fixed thresholdRange, not with the primary; a delta needs a custom MetricTemplate. Stages are stepWeight up to maxWeight, or a list such as stepWeights: [1, 2, 10, 80]. Flagger gives rollback time as interval × threshold, so its example of 1 minute and 10 leaves a failing canary serving for ten minutes, the "stopped after" column in the budget output.

Kayenta, Spinnaker's judge, does the statistics. In the 2026.1.3 source it runs a Mann-Whitney U test at 98% confidence. A metric must be statistically significant before its effect size is even checked, so the judge page notes that a 20% increase can still pass when the confidence interval is wide. Spinnaker's guidance asks for at least 50 data points per metric, and starting points of 3-hour canaries with a marginal score of 75 and a pass score of 95. One detail in the scorer: a metric group with no data at all scores 100, but if half or more of the metrics return no data, the whole canary fails.

A k6 probe for paths real traffic rarely reaches

The coupon path carried about 0.6% of the canary's traffic, so stage 2 saw 6 failures in a minute. The workbook does not count synthetic load as canarying, but it does suggest folding black-box checks into canary analysis. A k6 script aimed only at the canary pods, through the Rollout's canaryService, can push rare paths hard and quickly. Argo Rollouts treats a Kubernetes Job as a metric: exit code 0 passes, anything else fails.

// canary-probe.js: a black-box check sent only to the canary pods (an Argo Rollouts Job metric).
import http from 'k6/http';
import { check } from 'k6';

const TARGET = __ENV.CANARY_URL || 'http://127.0.0.1:3481';

export const options = {
  scenarios: {
    probe: {
      executor: 'constant-arrival-rate',
      rate: 20,
      timeUnit: '1s',
      duration: '1m',
      preAllocatedVUs: 10,
    },
  },
  thresholds: {
    'http_req_failed{step:coupon_order}': ['rate<0.01'],
    'http_reqs{step:coupon_order}': ['count>=100'],
    'http_req_duration{step:product}': ['p(95)<100'],
  },
};

export default function () {
  http.get(`${TARGET}/products/${Math.floor(Math.random() * 500)}`, { tags: { step: 'product' } });
  // Exercise the path the production mix reaches rarely: every probe iteration sends a coupon order.
  const res = http.post(`${TARGET}/orders`, JSON.stringify({ sku: 'SKU-1', qty: 1, coupon: 'PROBE-ONLY' }), {
    headers: { 'Content-Type': 'application/json' },
    tags: { step: 'coupon_order' },
  });
  check(res, { 'order placed': (r) => r.status === 201 }, { step: 'coupon_order' });
}

Against the defective canary it sent 1,201 coupon orders in a minute and exited 99:

    http_req_failed{step:coupon_order}
    ✗ 'rate<0.01' rate=100.00%

    http_reqs{step:coupon_order}
    ✓ 'count>=100' count=1201

level=error msg="thresholds on metrics 'http_req_failed{step:coupon_order}' have been crossed"

Against the healthy version it exited 0. The count floor is part 2's safeguard against a tag that receives no samples, whose thresholds pass. The probe has two costs. The Job metric fails on any non-zero exit, so an exception in the script (exit 107) aborts a good release exactly as a real failure (exit 99) does; failing closed is defensible because rollback is cheap, but choose it deliberately. And the probe writes real orders. The workbook warns that artificial load on billing paths has real side effects, so use a test tenant the business has agreed to, or probe read paths only.

Apply this to your next release

Check today whether your metrics can be split by version. Take one SLI with a version label, error ratio is the easiest, and write the comparison as canary against baseline in the same window, with a sample floor per cohort. Price your stage plan with the budget formula, using your real time to stop. Add an absolute SLO check on the control. Then send a deliberately broken build and a healthy one through staging under the same rules, and keep both decision records: a gate that has never aborted is untested. For which statuses count as failures, the HTTP status code reference helps; the workbook suggests leaving 4xx out of canary error counts and catching broken URLs with black-box probes.

Questions about canary analysis

What if the service gets too little traffic for these sample floors?

Lengthen the stage rather than lowering the floor, and use the probe for the paths you care about. The workbook ties the size and duration a canary needs to request rate and how varied requests are: the more uniform the traffic, the less you need.

Can two canaries run at once?

The workbook is direct: "We strongly advise running only one canary deployment at a time." Overlapping canaries contaminate each other's signals.

Should canaries run off-peak to be safe?

Off-peak lowers the cost of a bad release and its chance of being caught. The workbook notes that performance defects typically appear only under heavy load, so an off-peak canary can pass a release that fails at peak.

Primary references

• Canarying Releases, Google SRE workbook chapter 16. Canary and control, the 5% at 20% example, budget cost proportional to exposure, gradual stages, attributable metrics, the risk of before/after evaluation, shared failure domains, aggregation intervals no longer than the canary, one canary at a time, and the limits of artificial load.
• Alerting on SLOs, Google SRE workbook chapter 5. Burn rate, and the 14.4× one-hour fast-burn page used to show why service-wide alerts miss a small canary.
• Implementing SLOs, Google SRE workbook chapter 2. The error budget as 100% minus the SLO, and the error budget policy.
• Argo Rollouts canary strategy. setWeight and pause steps, replica-count rounding without traffic routing, and canaryService.
• Argo Rollouts analysis and progressive delivery. AnalysisRun phases and their effect on the rollout, successCondition and failureCondition, empty-result and NaN handling, Inconclusive runs, initialDelay, and the published error-rate example.
• Argo Rollouts Experiment CRD. Baseline and canary ReplicaSets from specRef, weighted experiment steps for SMI, ALB and Istio, and separate pod-template hashes.
• Argo Rollouts Job metrics. Exit code 0 passes, non-zero fails, and image-pull failures are Inconclusive.
• Kayenta integration page in the Argo Rollouts repo at v1.10.1. The example that passes the Stable and Latest hashes to the control and experiment scopes.
• Argo Rollouts analysis types at v1.10.1. inconclusiveLimit defaulting to 0.
• Argo Rollouts v1.10.1 release. Latest Argo Rollouts release when checked, published 2026-10-12.
• Flagger deployment strategies. stepWeight, maxWeight, stepWeights, and rollback time as interval × threshold.
• Flagger metrics analysis. The built-in success-rate and duration checks with fixed thresholdRange, and custom MetricTemplates.
• Flagger v1.45.0 release. The newest Flagger release on 2026-10-19.
• How the Spinnaker canary judge works. NetflixACAJudge: NaN handling, outlier removal, the Mann-Whitney U test at 98% confidence, effect-size thresholds and scoring.
• Canary best practices from Spinnaker. Comparing against an equivalent baseline deployed at the same time, at least 50 data points per metric, and starting thresholds of 75 and 95.
• Kayenta weighted-sum scorer at Spinnaker 2026.1.3. All-no-data groups scoring 100, and failure when half or more metrics return no data.
• NetflixACAJudge source at Spinnaker 2026.1.3. The classifier constructed with a 0.98 confidence level and 0.25 tolerance.
• PromQL operators. How or combines vectors, used to keep an absent error series from emptying the ratio.
• OpenTelemetry service attributes. service.version as the version string of the service.
• Grafana k6 thresholds guide. Thresholds as codified SLOs and the non-zero exit code when one fails.
• grafana/k6 at v2.3.0. The version used for the lab traffic and the probe.