Distributed log - Kafka from the partition up

Why this design

A distributed log is the answer to "how do twenty services find out that an order was placed, in order, without the order service calling each of them". Kafka, Redpanda, Pulsar and Kinesis all answer it the same way: write every event once to an append-only, partitioned, replicated log, and let each reader keep its own bookmark. The interview is rarely about the API. It is about whether you know that ordering is per partition, that an acknowledgement means "in the in-sync replica set" rather than "on a disk", and that a consumer which commits after processing will process some records twice.

What you will learn

• How hash(key) mod 96 picks a partition, and why that modulo makes the partition count a decision you make once.
• Why the producer batches for 5 ms into 64 KB batches, and how that turns 200,000 records/s into about 3,100 requests/s.
• What acks=all, min.insync.replicas=2 and the high watermark each guarantee - and the exact moment a lagging follower makes that guarantee weaker.
• Why followers PULL from the leader, and why the leader epoch exists so a returning ex-leader truncates its tail instead of keeping it.
• How a consumer group splits partitions, why it cannot split one, and what a rebalance costs a 96-partition group.
• The sizing: 240 MB/s in at peak, 720 MB/s of disk writes, 12 brokers, 104 TB local plus 41 TB tiered over 7 days.

How to read this board

1. Press play: eight steps follow one record from the producer's hash to a consumer's committed offset.
2. PRODUCE: the producer, the partitioner (a parallelogram, because it is a function inside the client) and the batch buffer.
3. ONE PARTITION: the leader, two followers - one of them in red because it fell out of the ISR - the high watermark diamond and the KRaft controller.
4. KEEP: segments, the tiered store, and compaction, which is a different retention model rather than a cleanup job.
5. CONSUME: the group coordinator, two consumers - the red one owns the hot partition - the offsets log and the dead-letter topic.
6. Pages 1 to 6 in order, then the DATA MODEL: brokers, topics, partitions, replicas, groups and committed offsets.

Ask these before drawing anything

• What must be ordered relative to what? "All events for one order" is a partition key. "All events, globally" is one partition and about 30 MB/s, which is a very different system. The answer is almost always per entity, and somebody has to say which entity.
• Is replay a requirement? If a new service must read the last 7 days, the log is the source of truth and retention is storage you are paying for on purpose. If not, a queue that deletes on acknowledge is cheaper.
• What does a duplicate cost? At-least-once is the default. If a duplicate charge is unacceptable, the consumer needs an idempotency key, and that is a table in the consumer's database, not a Kafka setting.
• What does a lost acknowledged message cost? This decides acks, min.insync.replicas and whether unclean leader election is ever allowed.
• How skewed are the keys? One tenant producing 40 percent of traffic turns a 96-partition topic into one partition and 95 idle ones.
• How fast must a consumer see a record? Batching and the high watermark both add latency; 10 ms end to end is achievable, 1 ms is not this design.

Functional

• Produce a keyed record to a topic; records with the same key are delivered in the order produced.
• Many independent consumer groups read the same topic, each at its own position.
• A consumer may rewind to any offset, or to a timestamp, inside the retention window.
• Retention by time (7 days) or by key (compaction keeps the latest value per key forever).
• Idempotent produce: a retried batch is never written twice.
• Transactions: a consume-transform-produce step either commits its output records and its input offsets together, or neither.

Non-functional

• 200,000 records/s sustained, 600,000/s peak, average record 1 KB.
• p99 produce latency of 15 ms at acks=all; p99 end-to-end (produce to consumer poll) of 50 ms.
• An acknowledged record survives the loss of any one broker, with no human involved.
• Losing one broker of three in a partition has no effect on writes; losing two stops writes to that partition rather than accepting them on one copy.
• Adding a consumer group costs disk reads on the brokers, not a change to any producer.

Out of scope

• Per-message routing, priorities or delayed delivery. A log has none of them; a queue built on top of it does.
• Exactly-once side effects outside the cluster. Transactions stop at the broker boundary.
• Global ordering across partitions.