Background job queue - at least once, and the retry that doubles it

Why this design

Almost every web application grows a background job queue: send the email after signup, generate the invoice PDF, call the webhook, resize the upload. It is the first place most engineers meet a distributed systems guarantee head on, because a worker can crash between doing the work and saying it did. Interviewers use it to see whether you know what "at least once" means, whether you can make a handler idempotent rather than hoping, and whether you have a plan for the job that fails every single time.

What you will learn

• Why exactly-once delivery is not on offer, and why at-least-once plus an idempotent handler is the honest substitute.
• Claiming with FOR UPDATE SKIP LOCKED, leases with a timeout, heartbeats for long jobs, and the reaper that returns abandoned work.
• Enqueueing in the same transaction as the business write, and the crash window it closes.
• Exponential backoff with jitter, a maximum attempt count, and why dead letters are a product feature.
• The one metric that says whether the queue is healthy: the age of the oldest ready job.

How to read this board

1. Press Play and follow the walkthrough: ENQUEUE on the left, THE QUEUE, the WORKERS, then SIDE EFFECTS on the right.
2. Then the failure paths: backoff and the reaper under THE QUEUE, DEAD LETTERS along the bottom.
3. Then OBSERVE on the left - the single number that pages someone.
4. 1 · Requirements to 6 · Deployment in the written band, then the DATA MODEL.
5. Then argue with the tension: the duplicate this design cannot prevent, and the primary it shares.

Ask these before drawing anything

• What do the jobs DO? A job that writes only to our own database can be made exactly-once with a transaction. A job that calls the outside world cannot, and the answer for each provider is different.
• How long does the longest job run? This sets the lease, and a lease shorter than the slowest job is a duplicate generator.
• Is order required? FIFO per customer is expensive and almost never needed. Ask for the actual business rule.
• What is the acceptable delay? "Within a second" and "within an hour" are different worker pools and different alerts.
• What happens to a job that will never succeed? Somebody must look at it. Who, and how quickly?

Functional

• Enqueue a job with a kind, a JSON payload, an optional run_at, a priority and an idempotency key, inside the caller's transaction.
• Workers claim ready jobs, run them, and mark them done; at least once delivery.
• Failed jobs retry with exponential backoff and jitter up to max_attempts, then move to dead letters.
• Abandoned jobs (worker died) are returned to ready when their lease expires.
• Dead letters can be inspected and redriven individually or in bulk after a fix.
• Scheduled and recurring jobs are enqueued by a cron enqueuer with run_at in the future.

Non-functional

• Pick-up latency: p99 under 2 seconds from ready to running at normal load.
• Throughput: 2,000 jobs a second sustained, with headroom to drain a one hour backlog in under 30 minutes.
• No job lost after the enqueueing transaction commits.
• Duplicates bounded to crashes inside the side-effect window, and every handler documents what a duplicate does.