Messaging & realtime
Presence - the most expensive cheap feature
heartbeats, TTL, and why one green dot costs more events per day than every message in the product
What this board gets wrong on purpose
The tension
Presence costs more than the product it decorates. Heartbeats are 400,000 writes a second at peak for a boolean that usually did not change, and a status change is fanned out to everyone currently looking at that name - 7.8 billion pushes a day, about eighty times the message volume of the chat product next to it. Every instinct for making it more accurate makes it worse: halving the TTL doubles the beat rate, and persisting the state produces a durable record that is wrong the moment it is written, because a backup of a set of live TCP connections is a lie. The board also draws no cache on the bulk read, so 180 million app launches a day each fetch 240 keys uncached - 500,000 reads a second whose only purpose is to colour some dots.
Requirements
Ask these before drawing anything
- How stale may a green dot be? Five seconds and five hundred milliseconds are the same feature to a user and a factor of ten in cost.
- Is presence per user or per device? "Online on desktop, away on mobile" is a different data model and a different fan-out from one bit per person.
- Who may see whom? Invisible mode, blocks and contacts-only turn every fan-out into a filtered fan-out, and filtering after the fan-out leaks.
- Do we need "away" and "busy", or only online and offline? Every extra state is another transition and another fan-out.
- Is last-seen a separate product promise? It survives everything else here and is the only presence fact worth putting on a disk.
- What is the largest number of people one person watches? The average is 240; the answer for a support agent with a 5,000-contact roster is what actually sizes the read path.
- What should happen during an outage - does everyone appear offline, or does everyone freeze at their last state? Both are defensible, and choosing afterwards means choosing badly.
- Is presence load-bearing for anything else? The moment routing, billing or notification suppression reads it, it stops being allowed to be wrong, and the whole design changes.
Functional
- Record that a device is alive, from a periodic heartbeat.
- Expire a device that stops beating, with no cooperation from it.
- Aggregate devices into one per-user state: online if any device is.
- Notify watchers when a user's state changes, subject to visibility rules.
- Answer a bulk query for the state of up to several hundred users at once.
- Fall back to "last seen at" when there is no live state.
- Support invisible mode, which must be indistinguishable from offline.
Non-functional
- A transition is visible to watchers within 5 seconds at p95. Not 500 ms - the extra nine costs an order of magnitude and nobody can see it.
- Cold-launch bulk query for 240 contacts: p95 under 400 ms.
- 99.9% availability, and presence is explicitly permitted to fail. It must degrade to last-seen rather than page anybody.
- Presence state is NOT durable and must not be. Rebuilding it from heartbeats takes one TTL interval.
- Invisible mode is a correctness requirement, not a best effort: leaking it once is a privacy incident.
Out of scope
- Typing indicators - same fan-out shape, ten times the event rate, and worth a board of their own.
- Location or activity presence ("listening to", "in a meeting").
- Using presence to route messages, which would make an eventually-consistent, deliberately-lossy system load-bearing for delivery.
The rest of this board
Available on Tier Slate
This page publishes the question. The answer — 5 more written pages, an 8-step narrated walkthrough and a 5-table schema — is the board itself, and it opens in Tierslate.
- 5written pages
- 8walkthrough steps
- 5tables23 columns
5 pages behind this one
- Capacity estimation
- Storage estimation
- Availability
- How it works
- Deployment plan
More boards
Delivery, presence, ordering, and who is typing.