Pastebin - a blob store with a short name

Why this design

Pastebin is a common early interview question because it looks like a URL shortener with a bigger value, and the interesting answer is the part that is not. Storing text is trivial: put it in an object store, keep a small row about it, hand back an id. The design is in what happens next - pastes expire, pastes are abused, and pastes are read from caches that do not know either of those things. The same shape appears in gists, shared snippets, crash-report uploads, temporary file shares and "share this config" links.

What you will learn

• Why the text goes in an object store and only ~300 bytes of metadata go in the database - and what that split costs.
• How to size it: 2M pastes a day is only ~23 writes/s, but 16 GB of new text a day and ~2,300 reads/s at peak.
• Why ids are random 8 character base62 strings (62^8 ≈ 2.2e14) rather than a counter: unlisted pastes must not be guessable.
• Lazy expiry on read against an expiry sweeper - and why you need both.
• Why immutable text is easy to cache and why that same cache defeats expiry, burn-after-reading and takedowns.

How to read this board

1. WRITE PATH across the top: author, client, the Paste API hex, the scan queue and the abuse scanner.
2. STORES on the right: paste metadata (the row), paste text (the blob) and the hot paste cache.
3. READ PATH across the bottom: reader, CDN edge, the read service hex, and the response going back.
4. TAKING IT AWAY on the right: the red expiry sweeper and moderation, the two things that try to make a paste unreadable.
5. Press play for the walkthrough, then read pages 1 to 6, then the DATA MODEL tables under the pages.
6. The conversation to have: every arrow that removes a paste is slower than the arrow that serves it. Decide which promise - expiry, burn-after-reading or takedown - you would fix first, and what it costs the cache.

Functional

• Create: POST text (up to 512 KB) with an optional title, syntax language, visibility (public or unlisted) and expiry (10 minutes, 1 hour, 1 day, 1 week, never, or burn after reading). Returns https://paste.example/{id}.
• Read: GET /{id} returns the paste rendered with syntax highlighting; GET /raw/{id} returns text/plain.
• Anonymous pastes are allowed; signed-in users can list and delete their own.
• An expired, deleted or taken-down paste returns 404 (or 410 Gone) - never the old text.
• Anyone can report a paste; moderators can take it down.

Non-functional

• Read latency: p99 under 100 ms for a cached paste, under 300 ms for one fetched from the object store.
• Read availability 99.95 percent; create availability 99.9 percent.
• Durability: a paste that was acknowledged is never lost before its expiry (object stores give 11 nines).
• Unlisted ids must be unguessable: no sequential ids, and no listing endpoint that reveals them.
• Expiry is a promise: an expired paste should be unreadable everywhere within a stated bound (say 5 minutes).
• Abuse controls: per-IP create rate limits, a size cap, and scanning for credentials and malware.

Out of scope

• Editing a paste in place. A new version is a new paste; immutability is what makes the cache simple.
• Full-text search across public pastes - a separate indexing pipeline, and an abuse magnet of its own.
• Collaboration, comments and forks.

Assumptions

New pastes per day                  2,000,000
Average paste size                       8 KB   (cap 512 KB)
Reads per paste, lifetime                  20
Reads per day      2M x 20       = 40,000,000
Peak factor        writes x 3, reads x 5

Requests per second

Writes   2M / 86,400 s        =     23 /s average
         x 3                  =    ~70 /s peak

Reads    40M / 86,400 s       =    463 /s average
         x 5                  = ~2,300 /s peak

One paste linked from a big forum thread
                              =  5,000 /s on ONE id

These are small numbers. A single modest database handles 70 inserts a second without noticing, and the read rate is well within one cache node. The numbers that matter are bytes, not requests - and the one viral paste, which is why there is a CDN at all.

Bandwidth

Ingress   23 /s x 8 KB        =   184 KB/s average
Egress    463 /s x 8 KB       =   3.7 MB/s average
          2,300 /s x 8 KB     =  18.4 MB/s peak  (~150 Mbit/s)
gzip on text, ~3:1            =   ~6 MB/s peak on the wire

Cache sizing

Most reads land in the first 24 hours of a paste
Pastes created per day         2M x 8 KB  =  16 GB
Cache the hottest 20 percent              = ~3.2 GB

A few gigabytes of hot text fits in one cache node with room to spare. It is replicated for availability, not for size.

Two things per paste

Metadata row (database)
  id 8 B, owner 16 B, blob key ~40 B, size, sha256 32 B,
  syntax, visibility, flags, created/expires 16 B
                                  ~150 B of data
  + index and row overhead        ~300 B per paste

Text (object store)               8 KB average

Growth

Text per day        2M x 8 KB          =   16 GB
Text per year       16 GB x 365        =  5.8 TB written
Never-expire share  40 percent
Retained per year   5.8 TB x 40%       =  2.3 TB

Metadata per year   730M x 300 B       =  219 GB written
Retained per year   x 40%              =   88 GB

This is why the text does not go in the database. Twenty-three terabytes of text after a decade is cheap in an object store (a few hundred dollars a month) and painful in a relational database, where it would bloat every backup and replica. The metadata, under a terabyte after ten years, fits one well-indexed Postgres primary with replicas.

Id space

Alphabet [0-9a-zA-Z]                 = 62
Length 8  62^8                       = 2.2e14
10 years of pastes                   = 7.3e9
Share of the space used              = 0.003 percent
Chance a random guess hits a paste   = 1 in 30,000

Random ids are generated by the API and inserted with a unique constraint; a collision (vanishingly rare) is simply retried. One in 30,000 is fine against a casual guess but not against a scraper making millions of requests, which is why reads are rate limited per IP too.

Dedup

Storing the SHA-256 of the text lets identical pastes share one blob, which matters for the abuse case: the same credential dump posted ten thousand times is one blob and one takedown. The cost is that a blob can only be deleted once no live row points at it.

What fails, and what it costs

• Object store. Regional, highly available, 11 nines of durability. If it is slow, uncached reads are slow; cached reads continue.
• Metadata database. One primary with two replicas. Losing the primary stops creates until failover (30-60 s); reads continue from replicas and cache.
• Hot paste cache. Losing it costs latency, never correctness - every miss falls through to the replica and the object store, which at 2,300 reads/s is well within their limits.
• CDN. If the edge is down or bypassed, the origin sees every read. Size the read service for the uncached peak.
• Scan queue. If it backs up, pastes are served unscanned for longer. Nothing breaks visibly, which is exactly why it needs an alert on queue age.
• Expiry sweeper (red). If it stops, nothing is ever deleted - storage grows and, if the read path forgets to check expires_at, expired text is served.

The write is two writes

The API writes the blob first, then the row. If it crashes between them, the blob is an orphan: harmless but billed. Writing the row first would be worse - a row pointing at nothing is a paste that 500s. So: blob first, row second, and a weekly job that lists blobs with no row older than a day and deletes them.

What degrades instead of dying

• Scanner down: pastes go live unscanned and are scanned when it recovers.
• Primary down: creates fail fast with a retryable error; reads are unaffected.
• Cache down: reads slow from ~5 ms to ~40 ms, nobody sees an error.

Write path

The author submits text. The Paste API checks the size cap and the caller’s rate limit, generates a random 8 character id, gzips the text and PUTs it to the object store under a key derived from its SHA-256. It then inserts the metadata row - id, blob key, size, syntax, visibility and expires_at - and returns the link. Finally it drops a scan job on the queue. The author has their link in about 50 ms; the scan happens after.

Read path

A reader opens the link. The CDN edge serves it if cached. On a miss the read service looks in the hot paste cache, and on a cache miss reads the row from a replica and the text from the object store. The read service checks the row on every origin read: status must be active and expires_at must be in the future, otherwise 404. Text is immutable, so the response carries Cache-Control: public, max-age=3600 - and that header is the whole tension.

Expiry, twice

• Lazy expiry: every origin read compares expires_at to now. This makes the origin correct the instant a paste expires.
• The sweeper: every few minutes it deletes rows WHERE expires_at < now() in batches of 10,000 using the index on expires_at, then deletes blobs no live row references. This reclaims space; it is not what makes expiry correct.
• The edge: neither of the above touches the CDN. Bound it by setting max-age to min(3600, seconds until expiry), and send Cache-Control: no-store for burn-after-reading.

Burn after reading

The first read must atomically flip the row to burned (UPDATE ... SET status = burned WHERE id = $1 AND status = active RETURNING blob_key) and only the request that wins that update gets the text. It must never be cached anywhere - not at the edge, not in the hot cache - and link-preview bots that fetch URLs pasted into chat will "read" it first unless the page requires a click to reveal.

Abuse

The scanner matches text against credential patterns (emails with passwords, API key formats), known-malware hashes and blocklisted domains, and marks the row flagged. Moderation takes a paste down by setting status, then must purge its URLs from the CDN. A takedown that skips the purge keeps serving for up to an hour.

Topology

• One region is enough for these numbers. Paste API and read service are stateless containers behind a load balancer, at least two per zone across two zones.
• Postgres primary with two replicas in different zones; reads go to replicas, with the author’s first read pinned to the primary so a fresh paste is never a 404.
• Object store bucket with versioning off and a lifecycle rule as a backstop: objects under the expiring/ prefix older than 8 days are deleted even if the sweeper fails.
• A managed CDN in front of the read service only. The API host is never cached.
• Sweeper and scanner as separate worker deployments, so a slow scanner never delays deletion.

Rollout

• Read service: canary 5 percent, watch p99, 404 rate and 5xx rate.
• Cache-Control changes ship behind a flag and start short: a header sent with max-age=86400 cannot be recalled from caches that already have it.
• Id length and alphabet are fixed forever - links are pasted into chat logs and bug trackers for years.

What to watch

Scan queue age (how long pastes are live unscanned), sweeper lag (oldest row past its expires_at), orphan blob count, CDN hit ratio, and create rate per IP - a sudden spike from a handful of addresses is somebody using you as free hosting.