A reconciliation job runs nightly at 02:30 in America/New_York. Over 2026 it fired 364 times, not 365. The missing night is 8 March, when 02:30 did not exist: the clock went from 01:59:59 EST straight to 03:00:00 EDT. Nothing failed. No alert fired, because nothing was watching for a run that was never scheduled. The day's ledger simply carried into the next run, and the discrepancy surfaced eleven weeks later in a finance query.
The same year, the 01:30 job on 1 November ran twice — once at 05:30 UTC and once at 06:30 UTC — because 01:30 local occurred twice that morning. Both behaviours are documented, both are correct, and neither is what the person who wrote 30 2 * * * had in mind.
Start with the rule that catches people before time zones ever enter the picture. The crontab(5) man page states it plainly: if both the day-of-month and day-of-week fields are restricted — that is, neither contains * — the command runs when either field matches. Its own example is 30 4 1,15 * 5, which runs at 04:30 on the 1st and 15th of each month, plus every Friday.
# ┌───────────── minute (0 - 59)\n# │ ┌─────────── hour (0 - 23)\n# │ │ ┌───────── day of month (1 - 31)\n# │ │ │ ┌─────── month (1 - 12)\n# │ │ │ │ ┌───── day of week (0 - 6 in Kubernetes; 0 - 7 in cronie, where 0 and 7 are both Sunday)\n# │ │ │ │ │\n 30 4 1,15 * 5 # NOT \"the 1st and 15th, if they are a Friday\"\n # It is \"the 1st, the 15th, AND every Friday\" — roughly 6 runs a month.Kubernetes inherits this exactly, because the CronJob controller vendors github.com/robfig/cron/v3, whose dayMatches returns domMatch && dowMatch when either field carries the star bit and domMatch || dowMatch otherwise. Same rule, same trap.
What Kubernetes does not inherit is the day-of-week range. Cronie documents 0–7 with both 0 and 7 meaning Sunday. The vendored Go parser declares dow = bounds{0, 6, ...}, so a schedule that has run for years on a Linux host is rejected outright when it is lifted into a manifest:
\"0 0 * * 7\" ERROR: end of range (7) above maximum (6): 7\n\"0 0 * * 0\" ok — Sun 2026-03-08 00:00 EST, Sun 2026-03-15 00:00 EDT, ...Two other dialects break the rule in opposite directions, and knowing which one you are holding matters more than knowing the syntax:
\n\nsystemd.time(7) gives Thu,Fri 2012-*-1,5 11:12:13 and glosses it as the first or fifth day of any month in 2012, \"but only if that day is a Thursday or Friday\". The weekday is a filter on the date, not an alternative to it.? means \"no specific value\" and is used when you need to specify one day field but not the other. So 0 9 * * 1 is Monday in cron, while the Quartz analogue 0 0 9 ? * 1 is Sunday.Every one of those differences disappears if you stop reviewing the expression and start computing the instants it produces. The unit under test is the scheduler's own parser, at the version you deploy, evaluated from a fixed base time.
\n\nFor anything running on Kubernetes, that means the vendored library:
\n\n// schedule_test.go — go test ./...\npackage schedule\n\nimport (\n\t\"testing\"\n\t\"time\"\n\n\tcron \"github.com/robfig/cron/v3\"\n)\n\n// Exactly the field set the CronJob controller parses: five fields plus macros.\nvar parser = cron.NewParser(\n\tcron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor,\n)\n\nfunc fireTimes(t *testing.T, spec string, from time.Time, n int) []time.Time {\n\tt.Helper()\n\ts, err := parser.Parse(spec)\n\tif err != nil {\n\t\tt.Fatalf(\"parse %q: %v\", spec, err)\n\t}\n\tout := make([]time.Time, 0, n)\n\tfor cur := from; len(out) < n; {\n\t\tcur = s.Next(cur)\n\t\tout = append(out, cur)\n\t}\n\treturn out\n}\n\nfunc TestNightlyReconcileSurvivesSpringForward(t *testing.T) {\n\tny, _ := time.LoadLocation(\"America/New_York\")\n\tfrom := time.Date(2026, 3, 7, 3, 0, 0, 0, ny) // just after the 7th has fired\n\n\t// robfig/cron accepts a TZ= prefix; .spec.schedule does NOT. In a manifest\n\t// this zone goes in .spec.timeZone instead. See the field notes below.\n\tgot := fireTimes(t, \"TZ=America/New_York 30 2 * * *\", from, 2)\n\n\t// This assertion FAILS, and that failure is the bug report.\n\twant := time.Date(2026, 3, 8, 2, 30, 0, 0, ny)\n\tif !got[0].Equal(want) {\n\t\tt.Fatalf(\"expected a run on 2026-03-08, next fire was %s\", got[0])\n\t}\n}Run it and the library tells you precisely what it intends to do:
\n\n--- next fires after 2026-03-07 03:00 from \"30 2 * * *\" in America/New_York\n 2026-03-09 02:30:00 EDT = 2026-03-09 06:30Z # 8 March never appears\n--- next fires after 2026-10-31 03:00 from \"30 1 * * *\"\n 2026-11-01 01:30:00 EDT = 2026-11-01 05:30Z\n 2026-11-01 01:30:00 EST = 2026-11-01 06:30Z # same wall clock, two instants\n 2026-11-02 01:30:00 EST = 2026-11-02 06:30ZThis is not library-specific misbehaviour. crontab(5) documents the identical outcome for classic cron: non-existent times \"will never match, causing jobs scheduled during the 'missing times' not to be run\", and times that occur more than once \"will cause matching jobs to be run twice\".
For systemd, the equivalent test needs no code at all, because systemd-analyze exposes the calendar engine directly and --base-time makes the answer deterministic:
# Does the timer really fire on the 29th of February, and when next?\nsystemd-analyze calendar --iterations=5 '*-2-29 0:0:0'\n\n# Pin the clock and ask for the next five elapses of a nightly job,\n# with the time zone written into the expression itself.\nsystemd-analyze calendar --iterations=5 \\\n --base-time='2026-03-07 12:00:00' \\\n 'Mon..Sun *-*-* 02:30:00 America/New_York'\n\n# Normalisation is a linting tool: the tool prints the expression it understood.\nsystemd-analyze calendar 'weekly Pacific/Auckland'\n# Original form: weekly Pacific/Auckland\n# Normalized form: Mon *-*-* 00:00:00 Pacific/AucklandReading the normalised form back is worth a habit of its own. It is the cheapest way to catch a schedule that parsed successfully and means something other than you intended.
\n\nDaylight-saving failures are not one bug. They are three, with different symptoms and different fixes.
\n\nThe gap: a run that never happens. Any wall-clock time inside the skipped hour has no instant, so it cannot match. In the United States in 2026 that hour is 02:00–02:59 on 8 March. Every schedule between 0 2 * * * and 59 2 * * * silently loses a day.
The fold: a run that happens twice. On 1 November 2026 the hour 01:00–01:59 occurs twice in America/New_York. An hourly job 0 * * * * fires 25 times that day, with 01:00 EDT and 01:00 EST both matching. If the job is not idempotent, this is the expensive one.
Midnight itself can vanish. The habit of moving a job to 0 0 * * * to dodge the 02:00 window works in North America and fails elsewhere. In America/Havana the spring transition happens at midnight, so 2026-03-08 has no 00:00 at all and a daily midnight job jumps 47 hours; on 1 November it fires twice. America/Santiago and Asia/Beirut transition at 00:00 too, in their own months.
Half-hour offsets are the sharp edge here, because scheduler DST handling is frequently written for whole hours. The vendored Go library corrects a mis-landed day by adding or subtracting one hour, an assumption its own source comment makes explicit. Point it at Australia/Lord_Howe, which shifts by 30 minutes, and a plain midnight job skips a day that exists:
# spec \"0 0 * * *\", zone Australia/Lord_Howe, robfig/cron v3.0.1\n2026-04-05 00:00 +11 -> 2026-04-07 00:00 +1030 delta 48h30m # 04-06 skipped\n2026-10-04 00:00 +1030 -> 2026-10-06 00:00 +11 delta 47h30m # 10-05 skipped\n\n# ...yet the skipped midnight is a perfectly real instant:\n2026-04-06 00:00:00 +1030 ->UTC 2026-04-05 13:30ZThe fix for all three is the same and it is boring: schedule in UTC and let the job decide what local day it is operating on.
\n\n# Wrong: the schedule carries the business time zone, so the schedule\n# inherits that zone's discontinuities.\nspec:\n schedule: \"30 2 * * *\"\n timeZone: \"America/New_York\"\n\n# Right: the schedule is a monotonic UTC instant. The job resolves\n# \"yesterday in New York\" itself, where a test can cover the edge cases.\nspec:\n schedule: \"30 7 * * *\"\n timeZone: \"Etc/UTC\"Kubernetes will not let you smuggle the zone in through the back door either: a TZ or CRON_TZ prefix inside .spec.schedule is not officially supported, and the API rejects the resource with a validation error. Use .spec.timeZone, stable since v1.27. With no zone set, the schedule is interpreted in kube-controller-manager's local zone — a property of the control plane, not of your manifest, and not something your repository can review.
Once the expression matches, three CronJob fields decide independently whether a Job object is actually created. Each has a distinct failure mode.
\n\n.spec.startingDeadlineSeconds is a deadline for a late start. The controller measures the gap between when a Job was expected and now; if the gap exceeds the limit, that execution is skipped, and Kubernetes treats it as a failed Job. Unset means no deadline. Below 10 seconds, the docs warn the CronJob may never be scheduled at all, because the controller only checks every 10 seconds.
.spec.concurrencyPolicy is Allow by default. Forbid skips the new run while the previous one is still going — and that skip counts as a missed schedule. Replace kills the running Job and starts a new one, which is the correct choice only for work that is safe to abandon mid-flight.
Then the rule that turns a temporary outage into a permanent one. The controller counts how many schedules were missed between the last scheduled time and now. Past 100, it refuses to start the Job and logs too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew. The documentation's own arithmetic: a job scheduled every minute from 08:30, with no starting deadline, whose controller is down from 08:29 to 10:21, does not start — more than 100 schedules were missed. Set startingDeadlineSeconds: 200 and the same outage ends with the job starting at 10:22, because the controller now counts misses only within the last 200 seconds.
concurrencyPolicy: Forbid with startingDeadlineSeconds unset, on a frequent schedule whose runtime occasionally exceeds its interval. Every blocked run is a missed schedule; enough of them in a row and the catch-up path stops firing. The manifest looks conservative and is the least safe of the three configurations.Kubernetes is explicit that scheduling is approximate: a CronJob creates a Job \"approximately once per execution time of its schedule\", and there are circumstances where two Jobs are created or none is. The documented instruction is to make the Job idempotent. Treat that as a testing requirement — run the job twice against the same input in CI and assert the second run is a no-op — rather than as a caveat.
\n\nSince v1.32, every Job a CronJob creates carries the annotation batch.kubernetes.io/cronjob-scheduled-timestamp, holding the originally scheduled creation time in RFC3339. That is the missing half of every \"did it run on time\" question: the schedule's intent, recorded next to the observed reality.
# Intended fire time vs. when the Job actually started, per run.\nkubectl get jobs -l batch.kubernetes.io/cronjob-name=nightly-reconcile \\\n -o jsonpath='{range .items[*]}{.metadata.annotations.batch\\.kubernetes\\.io/cronjob-scheduled-timestamp}{\"\\t\"}{.status.startTime}{\"\\n\"}{end}' \\\n | sort\n\n# 2026-03-07T07:30:00Z 2026-03-07T07:30:04Z\n# 2026-03-09T06:30:00Z 2026-03-09T06:31:57Z <- 8 March absent entirely\n\n# The controller-side symptom of the 100-missed rule.\nkubectl get events --field-selector reason=FailedNeedsStart -AAlert on the absence of an expected timestamp, not on job failure. A run that never started produces no failing pod, no non-zero exit and no log line. It produces a gap in this list, which is why the list has to exist before the gap does.
\n\nIf you are choosing between mechanisms, three documented differences decide it.
\n\nCatch-up is opt-in and coalesced. Persistent=true stores the last trigger time on disk; when the timer activates, the service is triggered immediately if it would have fired at least once while the timer was inactive. Once — not once per missed elapse. Kubernetes has no equivalent switch; its catch-up behaviour is whatever startingDeadlineSeconds and the 100-miss ceiling produce.
Timers are deliberately imprecise by default. AccuracySec= defaults to one minute, and the timer fires at a host-specific, randomised but stable point inside that window, to let the manager coalesce wakeups. If your test asserts second-level precision, set AccuracySec=1us or the test is asserting something the unit never promised.
Jitter is a first-class field. RandomizedDelaySec= defaults to 0 and spreads firing over an interval; FixedRandomDelay= makes that delay deterministic per machine, so a fleet stays spread out but each host keeps a stable slot. Reproducing an equivalent for a CronJob means adding a sleep inside the container, which is jitter your monitoring cannot distinguish from lateness.
A time zone is not a constant, and a test that hardcodes an offset is asserting a fact with an expiry date. The IANA database shipped three releases in 2025 and another three by July 2026. Release 2019b recorded that Brazil cancelled DST and would stay on standard time indefinitely — every schedule tuned around a São Paulo transition became wrong that year. Release 2026c, dated 8 July 2026, records Alberta moving to permanent −06 and Morocco moving to permanent UTC on 20 September 2026.
\n\nThree consequences for your tests. Assert on IANA zone identifiers and computed instants, never on literal UTC offsets or abbreviations like EST. Pin the tzdata version your test image uses, so a base-image bump that changes a fire time fails a test rather than a payment run. And check where the data actually comes from: Kubernetes embeds the Go standard library's copy as a fallback when no external database is available on the system, so the control plane and your job container can disagree about a zone that changed recently.
Take your most business-critical scheduled job today and do three things. Compute its next twenty fire times from a base time of 1 March, using the parser its scheduler actually uses, and check that the day of the spring transition appears. Add one test that asserts the run count over the DST week — 7 for a daily job, 167 or 169 for an hourly one, never 168. Then run kubectl get jobs with the cronjob-scheduled-timestamp annotation and diff intended against actual for the last thirty runs.
The evidence to keep is that intended-versus-actual list. It converts \"the job seems fine\" into a number, and it is the only artefact that shows a run which did not happen.
\n\nThe scheduling problem is. UTC has no gaps or folds, so a UTC schedule fires the same number of times every week forever. What moves is the business logic: a report due at 09:00 local now arrives an hour early or late for half the year. Either accept the drift, or have the job compute the local target itself and exit early when it is not yet due — a decision that lives in code you can unit test, rather than in a five-field string you cannot.
\n\nUp to a point. With startingDeadlineSeconds unset or large and concurrencyPolicy: Allow, the documentation says Jobs will always run at least once. But the controller stops starting the Job once it counts more than 100 missed schedules since the last scheduled time. Setting startingDeadlineSeconds changes the counting window to that many seconds back, which is how you avoid tripping the ceiling.
? work in a Kubernetes schedule?Yes, and it means something different from Quartz. Kubernetes documents ? as having the same meaning as * — any available value for that field. In Quartz it means \"no specific value\" and exists so you can leave one of the two day fields unspecified. Copying a Quartz expression into a manifest also loses the leading seconds field and shifts every value one position left.
Do not manipulate the system clock. Pass an explicit base time to the scheduler's own calculator: --base-time for systemd-analyze calendar, or a fixed time.Time into Schedule.Next in Go. Both compute future elapses from any starting instant, which makes a DST test as fast and deterministic as any other unit test.
The job, not the schedule. Kubernetes states outright that two Jobs may be created for one execution time and that yours should therefore be idempotent, and the autumn fold produces the same duplicate on a plain Linux host. A schedule change removes one cause of a double run; a natural key or a scheduled-time-derived idempotency token removes all of them. The cronjob-scheduled-timestamp annotation is a ready-made token for exactly this.
30 4 1,15 * 5 example, day-of-week 0–7, and the statement that missing DST hours never match while repeated hours run jobs twice? as a synonym for *, startingDeadlineSeconds, the three concurrencyPolicy values, the 100-missed-schedule rule with the 08:29–10:21 worked example, .spec.timeZone, rejection of TZ/CRON_TZ in the schedule, approximate job creation, and the batch.kubernetes.io/cronjob-scheduled-timestamp annotationdayMatches implementing OR, dow = bounds{0, 6}, and the whole-hour DST correction in SpecSchedule.NextPersistent= catch-up semantics, AccuracySec= defaulting to one minute, RandomizedDelaySec= defaulting to zero, and FixedRandomDelay=calendar verb, --iterations=, and --base-time=? as \"no specific value\"