Infrastructure primitives
Secrets manager - the one store everything trusts
How secrets are encrypted under a key nobody holds directly, and why rotation is the feature, not storage.
What this board gets wrong on purpose
The tension
Everything trusts this store, so everything waits on it: a pod cannot start until it has read its secrets, and the agent's five-minute cache helps pods that are already running and does nothing for a new one. A KMS throttle or a secrets-API outage during an autoscale or a full-fleet restart - 12,000 pods, 72,000 reads in ten minutes - means nothing new comes up, which turns a vault incident into an every-service incident. Rotation is drawn correctly on the vault's side (two database users, alternate which one is changed, stage then test then promote) and not at all on the consumer's: a service that read its password once at boot into a config object keeps working after the first rotation because the other user is untouched, and fails thirty days later when that user's password changes - an outage dated a month after the deploy that caused it. Plaintext also outlives every policy here, cached in agent memory and in environment variables that crash dumps happily print.
Practise this as an interview
Take Secrets manager - the one store everything trusts as a timed system design interview on a blank board - on your own, with the requirements to check yourself against, or with a candidate you invite.
Start here
Why this design
Every company eventually discovers its database passwords in a Git history, a CI log and a Slack message, and buys or builds a secrets manager. The interesting part is that the obvious version - an encrypted table and an API - solves the least important problem. A secrets manager is really three systems: envelope encryption so that holding the data is not holding the secrets, machine identity so that a workload can prove who it is without a password, and rotation so that a leaked credential stops working on a schedule nobody has to remember.
What you will learn
- Envelope encryption: a data key per secret version, wrapped by a key-encryption key that never leaves the HSM - and why that makes a stolen backup worthless.
- Why "authenticate with a secret to get your secrets" is a loop, and how platform-issued workload identity breaks it.
- How dual-user rotation changes a database password under a live fleet with no outage, step by step.
- Why the read rate is set by deploys and cold starts rather than by traffic, and what a 12,000-pod restart does to the KMS quota.
- Why the audit log is the actual product for a security team, and why it is larger than everything else combined.
How to read this board
- Press Play: the walkthrough follows a pod booting, proving who it is, and receiving a password decrypted under a key nobody holds - then a rotation changing that password under it.
- CALLERS (left) is who is asking; THE VAULT (middle) is ciphertext plus a KMS drawn red; ROTATION (right) is the feature.
- 1 · Requirements, 2 · Capacity and 3 · Storage - including why the audit log outweighs the secrets 10,000 to one.
- 4 · Availability - why the vault is on the boot path of every service, and how to get it off.
- 5 · How it works in prose, then 6 · Deployment - including the root key ceremony nobody wants to think about.
- Finally the DATA MODEL, then argue with the weakness: rotation is drawn on the vault's side and not on the consumer's.
1 · Requirements
Ask these before drawing anything
- Which secrets are static (an API key a vendor issued) and which can be rotated by us (a database role we own)? Only the second kind can be rotated automatically, and it is usually the minority.
- How do workloads identify themselves today - Kubernetes service accounts, cloud IAM roles, VM instance identity, or nothing? The answer decides whether this is a weekend or a year.
- Is a secret read once at boot, or on every use? The first is cheap and makes rotation dangerous; the second is safe and puts the vault on the request path.
- What compliance regime applies? PCI and SOC 2 set audit retention (a year is common) and require proving who read what - which sizes the largest table here.
- Who can read a secret as a human, and how is break-glass access granted and reviewed?
- Should the vault issue dynamic credentials (a fresh database user per lease) rather than store static ones? It removes rotation entirely and adds a database write per lease.
Functional
- Store versioned secrets under hierarchical paths (prod/payments/db/orders).
- Encrypt every version under its own data key, wrapped by a KMS key-encryption key.
- Authenticate workloads by platform-issued identity; authorise by policy over identity and path.
- Rotate rotatable secrets on a schedule through stages: pending, current, previous.
- Log every read, write, denial and rotation step, immutable.
- Soft-delete with a recovery window; never hard-delete a version still referenced.
Non-functional
- Read p99 under 30 ms including the KMS unwrap.
- Availability 99.99% - it is on the boot path of every service.
- A full dump of the database plus every backup must not reveal a single plaintext.
- No plaintext at rest anywhere the vault controls; plaintext in transit only over mTLS.
- Rotation of a dual-user secret completes with zero failed authentications on consumers that re-read.
Out of scope
- Certificate issuance (a PKI is a neighbouring system with its own board).
- Customer-managed keys for tenant data - the same envelope pattern, a different product.
- Detecting secrets committed to Git - a scanner, upstream of this.
The rest of this board
Available on Tier Slate
This page publishes the question. The answer — 5 more written pages, an 8-step narrated walkthrough and a 6-table schema — is the board itself, and it opens in Tierslate.
- 5written pages
- 8walkthrough steps
- 6tables40 columns
5 pages behind this one
- Capacity estimation
- Storage estimation
- Availability
- How it works
- Deployment plan
More boards
The building blocks an interview asks you to build from scratch.
- URL shortener - one write, a million reads
- Distributed cache - the ring, and what a node leaving costs
- Key-value store - quorums, and where the consistency knob sits
- Rate limiter - token bucket, and where the counter lives
- Unique ID generator - 64 bits, and two facts nobody can check
- Web crawler - a frontier, and the trap of crawling forever
- Pastebin - a blob store with a short name
- Load balancer - spreading traffic, and noticing who is dead
- API gateway - one front door for many services
- Login and sessions - who you are, on every request
- Feature flags - a config change that ships like code
- DNS - the cache that runs the internet
- Distributed cron - exactly one runner at nine o'clock
- Background job queue - at least once, and the retry that doubles it
- Code execution sandbox - running a stranger's program
- Distributed log - Kafka from the partition up
- Consensus - a leader, a log, and a majority
- Object storage - eleven nines from cheap disks