Analytics pipeline - the lag between an event and the answer

Ask these before drawing anything

• How fresh does the answer have to be, and who says so? "Real time" is never the requirement. The requirement is a person with a decision, and the honest question is how much later than the event that decision may be made. Ten seconds and one hour are different architectures, not different settings.
• Is the number allowed to change after it is shown? If yes, the stream path is viable and the reconciliation job below is mandatory. If no, there is only a batch path and this board gets simpler.
• What is the cost of a duplicate, and what is the cost of a miss? Double-counting a page view is a rounding error. Double-counting a purchase is a finance incident. The answer decides how much of the exactly-once machinery is worth paying for.
• How late may an event be and still count? This single number sets how long every time bucket stays mutable, and therefore how long results cannot be cached, archived or trusted.
• Who owns the event schema - the pipeline team or the mobile team that emits it? Whoever owns it decides whether a renamed field is a deploy or an outage.
• Are these events personal data? If they are, retention is not a cost decision any more, deletion has to be possible by subject, and immutable object storage becomes a liability.

Functional

• Accept events from web, mobile and server SDKs, at-least-once, with an idempotency key on every event.
• Validate each event against a registered schema and reject - loudly, to a quarantine topic - what does not fit.
• Keep every raw event exactly as received, before any processing, so that any future question can be asked of the past.
• Compute per-minute and per-hour rollups over a dozen dimensions, visible within ten seconds of the event.
• Recompute the same rollups nightly from the raw lake and replace the streaming values where they differ.
• Accept events up to 24 hours late and fold them into the bucket they belong to, not the bucket they arrived in.
• Serve arbitrary time-range queries over the rollups in under 500 ms at p95.

Non-functional

• Ingest availability 99.95 percent. The collector is the only part a client talks to, and a rejected event is gone - the SDK buffers, but only for a while and only on a device that stays open.
• End-to-end streaming lag under 10 seconds at p95, under 60 seconds at p99.
• No event loss once acknowledged. Acknowledgement happens after the log write is replicated, not before, and that choice costs about 8 ms of collector latency.
• The batch answer is authoritative. Where batch and stream disagree, batch wins, and the disagreement is measured rather than assumed.
• Cost per event must stay under one tenth of a cent, which at 200 million events a day is what makes retention a design decision rather than a default.

Out of scope

• Session stitching and identity resolution. Both belong upstream of this board and both are harder than anything on it.
• Machine learning features. The lake feeds them, but a feature store has different freshness and different correctness requirements.
• Sub-second answers. Anything that needs to react inside one second is not analytics, it is the serving path of the product, and it must not be built on this.