Distributed cache - the ring, and what a node leaving costs

Ask these before drawing anything

• What is being cached, and what does a stale answer cost? A stale product description is nothing; a stale permission check is a security incident. This answer decides whether TTL alone is acceptable or whether explicit invalidation is mandatory.
• Is this a cache or a store? A cache may lose everything and the system still works, slowly. If losing it loses data, it is not a cache and consistent hashing is the wrong tool - use the key-value store board instead.
• What is the working set, and what is the total key space? Caching is only economic when a small fraction of keys serves a large fraction of reads. If access is uniform, a cache is just expensive RAM.
• How big is the largest value, and how hot is the hottest key? Both have answers that break the design, and both are usually unknown until somebody measures.
• Who invalidates - the writer, a TTL, or nobody? Nobody is a valid answer and should be said out loud rather than discovered.
• Can the origin survive a total cache loss? If not, the cache is load bearing, and the availability requirement is really the origin capacity requirement.

Functional

• get, set and delete on an opaque byte value, keyed by an opaque string.
• A per-key TTL, set by the caller, defaulting per namespace.
• Read-through on a miss, with exactly one fill per key regardless of how many callers missed at once.
• Add and remove nodes without restarting clients and without moving more than one node share of the keyspace.
• Report per-key request counts well enough to identify a hot key while it is hot, not the next morning.

Non-functional

• p99 of 1 ms for a hit, measured at the client including the network. If a cache is not a great deal faster than the origin it is a liability rather than an optimisation.
• Hit rate at or above 95 percent in steady state. Everything below that arrives at a database sized for 5 percent.
• 99.9 percent availability for the cache, and a system that stays UP at 0 percent cache availability. Those two numbers are different requirements and the second is the important one.
• Adding or removing a node moves at most 1/N of keys and needs no client restart.
• The cache never becomes the source of truth. Any value in it can be recomputed from the origin.

Out of scope

• Durability. Nothing here is written to disk, and that is a feature - it is why a node restart is quick and why the capacity maths is only about RAM.
• Cross-region replication of cache contents. Each region caches for itself; see the deployment page.
• Transactions across keys. A cache with transactions is a database wearing a disguise.