Distributed cron - exactly one runner at nine o'clock

Why this design

Every company starts with a crontab on one box and eventually loses a month-end report because that box was being patched at midnight. The next step - the same crontab on three boxes - sends every customer three invoices. A distributed scheduler is the answer, and it is a good interview question because the obvious fix (leader election) is necessary, insufficient, and comes with a failure mode that only a fencing token handles. Then the real question arrives: the job ran once, but did it happen once?

What you will learn

• How a lease with a fencing token elects one ticking scheduler, and why the lease alone still allows two leaders for a few seconds.
• Why a unique constraint on (job_id, scheduled_for) is the actual exactly-once mechanism for scheduling.
• Misfire policies - skip, once, catch up - and why each job needs its own.
• Why execution is at-least-once no matter what the scheduler does, and what idempotent jobs look like.
• Why the top of the hour is a thundering herd 630 times the average rate, and how jitter flattens it.

How to read this board

1. Press Play: the walkthrough follows the 09:00 invoice job from its definition through the lease, the claim and the queue, then breaks it twice.
2. DEFINITIONS (left) is what should run and the alarm for what did not; THE TICK (middle) is one leader and one row per run; EXECUTION (right) is where exactly-once ends.
3. 1 · Requirements, 2 · Capacity and 3 · Storage - including the 33,000-run second at midnight.
4. 4 · Availability - a scheduler outage is silent, which is why the dead-man's switch is on the board.
5. 5 · How it works in prose, then 6 · Deployment - including how to deploy the scheduler without missing a tick.
6. Finally the DATA MODEL, then argue with the weakness: scheduling is exactly-once and execution is not.

Ask these before drawing anything

• What precision is needed - to the second, or within a minute? Second-level precision at scale means a leader scanning every second; minute-level allows a much lazier design.
• What should happen to a run missed during an outage? A report wants one late run; a cache warmer wants none; a metering job wants every interval it missed.
• May two runs of the same job overlap if one runs long? Forbid, allow, or replace the old one - another per-job field.
• Which timezone is "nine o'clock"? A customer's local nine moves twice a year relative to UTC, and on two days a year some local times do not exist or happen twice.
• Are the jobs ours or the customers'? A multi-tenant scheduler needs per-tenant limits, or one tenant scheduling 10,000 jobs at midnight starves everyone.
• Is the job itself idempotent? If not, no scheduler design delivers exactly-once, and the requirement belongs to the job owner.

Functional

• Register jobs with a cron expression, a timezone, a misfire policy, an overlap policy and a maximum runtime.
• Fire each scheduled occurrence once, on time, from exactly one scheduler.
• Dispatch each run to a worker, track it by heartbeat, and retry runs whose worker disappeared.
• Hand every run a fencing token that downstream writes can check.
• Alert when an expected successful run has not happened by its deadline.

Non-functional

• Fire time: within 2 s of the scheduled second at p99, outside the top-of-hour herd.
• Leader failover: under 15 s, with missed ticks handled by policy rather than lost.
• No duplicate run rows, ever - enforced by the database, not by code.
• Scheduler down for an hour must not silently drop an hour of runs.

Out of scope

• Workflow DAGs (job B after job A succeeds) - that is an orchestrator built on top of this.
• Sub-second timers and per-user delayed messages - a delay queue, not cron.
• The job code itself, beyond the contract that it must be idempotent.