Infrastructure primitives
Consensus - a leader, a log, and a majority
How Raft keeps replicas agreeing through elections and a replicated log, and what a network partition really does.
What this board gets wrong on purpose
The tension
Every write is ordered by one leader and waits for a majority to fsync it, so throughput is one machine's and latency is the round trip to the third-fastest of five disks - placing the five across three regions for survivability turns a 2 ms write into a 70 ms one. When the leader dies nothing can be written for an election timeout plus a round of voting, typically 300 ms to 2 s, and setting the timeout lower to shrink that makes a GC pause or a congested link trigger needless elections. A partition leaves the minority side completely unable to commit; an old leader stranded there keeps accepting writes it can never commit, so its clients get timeouts whose outcome is genuinely unknown and must be retried with a session id or risk applying twice. Reads are worse than they look: a leader answering from its own state machine can serve a value the majority already overwrote, and the two fixes cost either a heartbeat round per read (ReadIndex) or a bound on clock drift between machines (leader leases) - a timing assumption in an algorithm otherwise chosen because it makes none. Five nodes tolerate two failures and six still tolerate only two, so an even-sized cluster buys cost and nothing else; and changing membership one node at a time is safe only if every step completes, so a half-finished change during an outage is the classic way to lose quorum permanently.
Practise this as an interview
Take Consensus - a leader, a log, and a majority as a timed system design interview on a blank board - on your own, with the requirements to check yourself against, or with a candidate you invite.
Start here
Why this design
etcd under Kubernetes, Consul, CockroachDB and TiKV ranges, Kafka's KRaft controller and MongoDB's replica sets all run some form of Raft. It is how a small group of machines agrees on one ordered history despite crashes and partitions, and it sits under every system that needs "exactly one leader" or "this config change happened, everywhere, once". The interview question is usually "design a strongly consistent key-value store" or "how does leader election work", and the depth is in three places: why a majority is the magic number, what a partition does to each side, and why a read is harder than a write.
What you will learn
- Terms, randomised election timeouts, and the one vote rule that guarantees a new leader holds every committed entry.
- AppendEntries and its consistency check (prevLogIndex, prevLogTerm), and how a follower's divergent tail is cut off.
- Why commit means "on a majority's disk", and why a leader may only count replicas for entries from its own term.
- Exactly what each side of a 3-2 partition can and cannot do, including the stranded old leader.
- PreVote, and why without it a node that rejoins a cluster can force a healthy leader to step down.
- ReadIndex versus leader leases, and what each assumes.
- The sizing: 5 voters, 5,000 writes/s, a 2.4 ms write in one region, an 8 GB state machine and snapshots every 13 minutes.
How to read this board
- Press play: eight steps follow one write to commit, then the partition, the election that cannot finish, and a read that must not be stale.
- CLIENT: the caller, the session id that makes retries safe, and ReadIndex.
- THE LEADER: one writer for term 7, its log, the majority check and the state machine.
- MAJORITY SIDE and MINORITY SIDE: the same five nodes, cut 3-2 by a partition. Red is the side that cannot commit.
- ELECTION: the timer, the vote rule and PreVote. Then snapshots and membership change.
- Pages 1 to 6, then the DATA MODEL: the state each node must persist, as tables.
1 · Requirements
Ask these before drawing anything
- What is replicated? Cluster metadata (thousands of writes a second, gigabytes) suits one Raft group. A database (millions of writes, terabytes) needs thousands of groups, one per key range.
- How many failures must it survive, and of what? Two simultaneous node failures needs five voters. A whole region needs voters in three regions and pays cross-region latency on every write.
- Must reads be linearizable? If a reader may see a value up to a second stale, followers can serve reads and the leader is not the read bottleneck. If not, every read goes through the leader with a confirmation.
- What does an ambiguous write cost? A client that times out does not know whether its write committed. The answer decides whether you need client sessions.
- How large is the state? It decides snapshot size, and so how long a replacement node takes to catch up.
Functional
- Linearizable put, get, delete and compare-and-swap on keys.
- Watch: stream every committed change after a revision.
- Leases: keys that expire unless renewed, which is how leader election for OTHER services is built on top.
- Add and remove members online.
- A client retrying an ambiguous write never applies it twice.
Non-functional
- Survive any 2 of 5 node failures with no loss of committed data.
- p99 write under 10 ms in one region; reads under 5 ms linearizable.
- 5,000 writes/s and 50,000 reads/s sustained on one group.
- Leader failover under 2 s with no human involved.
- Safety under every partition, message loss, reordering and duplication. Liveness only when a majority can talk.
Raft promises safety always and progress only when a majority is connected. That asymmetry is the whole contract: it would rather stop than be wrong.
Out of scope
- Byzantine faults. Raft trusts every node to follow the protocol; a node lying about its log breaks it.
- Cross-group transactions. Two Raft groups agreeing needs two-phase commit on top.
The rest of this board
Available on Tier Slate
This page publishes the question. The answer — 5 more written pages, an 8-step narrated walkthrough and a 6-table schema — is the board itself, and it opens in Tierslate.
- 5written pages
- 8walkthrough steps
- 6tables39 columns
5 pages behind this one
- Capacity estimation
- Storage estimation
- Availability
- How it works
- Deployment plan
More boards
The building blocks an interview asks you to build from scratch.
- URL shortener - one write, a million reads
- Distributed cache - the ring, and what a node leaving costs
- Key-value store - quorums, and where the consistency knob sits
- Rate limiter - token bucket, and where the counter lives
- Unique ID generator - 64 bits, and two facts nobody can check
- Web crawler - a frontier, and the trap of crawling forever
- Pastebin - a blob store with a short name
- Load balancer - spreading traffic, and noticing who is dead
- API gateway - one front door for many services
- Login and sessions - who you are, on every request
- Feature flags - a config change that ships like code
- Secrets manager - the one store everything trusts
- DNS - the cache that runs the internet
- Distributed cron - exactly one runner at nine o'clock
- Background job queue - at least once, and the retry that doubles it
- Code execution sandbox - running a stranger's program
- Distributed log - Kafka from the partition up
- Object storage - eleven nines from cheap disks