Messaging & realtime
Push notifications - the token outlives the install
APNs and FCM, token lifecycle, batching, and why a dead token beats a failed send to the top of the incident list
What this board gets wrong on purpose
The tension
Delivery is measured at the provider's door, not at the phone. A send to a token belonging to a deleted app returns 200 and is counted as delivered, so a cohort can go completely dark while the dashboard stays at 99.9% - and nothing on the board can tell "delivered" from "accepted by a cloud that will drop it". The only correction is the feedback band, which is asynchronous, lossy, differently shaped per provider, and by far the least-exercised code here. It is also the most dangerous: the reaper retires tokens on a signal nobody verifies, so one mis-parsed 410 permanently un-notifies a real user with no error anywhere. And the batcher, which exists so a 25M broadcast does not open 25M connections, delays every transactional notification behind whatever marketing is sending.
Requirements
Ask these before drawing anything
- Are these notifications transactional, marketing, or both? If both, they must not share a queue, and the reason is on the capacity page.
- What is the latency promise? "A 2FA code within 5 seconds" and "a digest sometime this evening" are two systems, and batching is fatal to the first.
- Do we need to know whether a notification was SEEN, or only that it was accepted? Seen requires client telemetry and changes the data model.
- How do we define a dead token, and how long may a token stay dead before we notice? This is the number the whole board is built around.
- Can the payload contain personal data? If yes it crosses a third-party cloud, which is a compliance question before it is an engineering one.
- Is there a per-user rate limit or a daily cap? Without one, every team with API access can burn the notification channel for everyone else.
- Web push and email fallback: are they in the same system? Sharing preferences across channels is easy; sharing delivery is not.
- What happens when the provider is down for an hour - do we drop, delay, or deliver late? A stale notification can be worse than none.
Functional
- Accept a send for a user and deliver it to every eligible device they own.
- Respect per-category preferences, quiet hours and mutes; suppress when the user has already seen the thing on another surface.
- Collapse repeated notifications for the same subject into one (collapse keys).
- Retry transient provider failures with backoff; never retry a permanent one.
- Detect and retire dead tokens, and stop sending to them.
- Register, rotate and de-duplicate device tokens.
- Report, per notification, what happened as far as we can honestly know.
Non-functional
- Transactional: p95 from accept to handed-to-provider under 2 seconds.
- Marketing: a 25M-user broadcast fully handed over within 30 minutes.
- 99.9% availability for accepting a send. Delivery itself is explicitly not ours to promise, and the SLA says so.
- At-most-once per collapse key per window; duplicates are worse than lateness here, because a duplicate is visible on a lock screen.
- Dead tokens retired within 24 hours of the provider telling us.
Out of scope
- Composing the copy, localisation, and A/B assignment - upstream of the API.
- In-app inbox and web push.
- Deciding WHETHER something is worth notifying about, which is a product problem that no amount of infrastructure fixes.
The rest of this board
Available on Tier Slate
This page publishes the question. The answer — 5 more written pages, an 8-step narrated walkthrough and a 5-table schema — is the board itself, and it opens in Tierslate.
- 5written pages
- 8walkthrough steps
- 5tables39 columns
5 pages behind this one
- Capacity estimation
- Storage estimation
- Availability
- How it works
- Deployment plan
More boards
Delivery, presence, ordering, and who is typing.