Object storage - eleven nines from cheap disks

Why this design

S3, Google Cloud Storage, Azure Blob, Ceph RGW and MinIO all sell the same promise - put bytes under a key, get them back, never lose them - and all keep it the same way: separate the small, strongly consistent metadata (which keys exist, where their pieces are) from the enormous, append-mostly data, then cut the data into fragments with an erasure code and spread them so no single failure takes enough. The interview question "design S3" is answered well by someone who can do the durability arithmetic, explain why repair speed matters more than the code, and say what the metadata tier costs.

What you will learn

• Why 3x replication and RS(9,6) survive the same zone-plus-disk failure, at 3x and 1.67x the data.
• How placement spreads 15 fragments over 3 zones and 15 racks, and why that rule matters more than the code.
• The durability arithmetic: about 11 nines for 3 copies with a 6-hour repair, about 26 for RS(9,6) - and why nobody believes either.
• Why the manifest write is the commit point, and how that gives strong read-after-write.
• Scrubbing, prioritised repair, and why rebuilding one fragment reads nine.
• Why small objects are packed into volumes, and what that does to DELETE.
• The sizing: 100 PB logical, 50 billion objects, about 10,400 disks instead of 18,750, and four disk failures a week.

How to read this board

1. Press play: eight steps follow one 24 MB upload into fifteen fragments, then a disk failure and its repair, then a small object.
2. ENTRY: the application, the API front end, and versioning - the defence against bugs rather than disks.
3. METADATA: the index that decides whether an object exists, its key-range partitions, and lifecycle.
4. DATA: the erasure coder, the placement decision, the small-object packer, and three zones - the red one has lost a disk.
5. KEEPING ELEVEN NINES: scrubber, repair queue, rebuilder, garbage collector.
6. Pages 1 to 6, then the DATA MODEL.

Ask these before drawing anything

• What durability is promised, and per what? "Eleven nines per object per year" means an expected loss of one object in a hundred billion per year. Write it down; every other number follows from it.
• Must the store survive a whole zone? If yes, every write crosses zones and the code must be chosen so one zone holds no more fragments than the code can lose.
• What is the size distribution? Median objects in most buckets are under 100 KB while most BYTES are in objects over 10 MB. The small ones decide request costs and the large ones decide disk costs.
• Is read-after-write consistency required, and for LIST too? It decides how the metadata tier is built.
• What is the read pattern? Write once, read rarely (backups) wants cold encodings; hot media wants caches in front.
• Is deletion legally required to be prompt? It conflicts with delayed garbage collection, which is a safety net.

Functional

• PUT, GET (including byte ranges), HEAD, DELETE, and LIST by prefix with pagination.
• Multipart upload for objects up to 5 TB in parts of 5 MB to 5 GB.
• Versioning, delete markers, and object lock (write once, read many) per bucket.
• Lifecycle rules: move to a cold tier after N days, expire after M.
• Strong read-after-write and list-after-write consistency.

Non-functional

• 99.999999999 percent annual durability per object; 99.99 percent availability.
• Survive the loss of one availability zone with no data loss and continued reads and writes.
• GET first byte p99 under 100 ms for hot objects; PUT p99 under 200 ms for 1 MB.
• 1,000,000 GET/s and 100,000 PUT/s across the region.
• Storage cost: under 1.7 raw bytes per logical byte for objects over 1 MB.

Out of scope

• POSIX semantics - rename, append, partial overwrite. An object is replaced whole.
• Cross-region replication (it is an asynchronous copy job on top of this design).
• A CDN; caching in front of GET is a separate tier.