A/B testing platform - assignment is the easy half

Why this design

Every product team that ships changes and wants to know whether they helped needs an experimentation platform, and the first version is usually a random number stored per user and a dashboard counting purchases per arm. Interviewers ask this to see whether you know that deterministic hashing makes assignment trivially scalable, and - far more importantly - whether you understand what makes a result trustworthy: exposure at point of use, events after exposure, user-level aggregation, sample ratio checks and honest handling of peeking. Data teams that run these platforms spend almost none of their time on bucketing and almost all of it on the metric pipeline.

What you will learn

• Why hash(salt + user_id) mod 10,000 replaces a stored assignment, and how salts and layers keep experiments independent or exclusive.
• Why exposure is logged when a variant is used, and how first-exposure is computed.
• Why the metric pipeline joins events to exposure by time and aggregates per user before comparing arms.
• What a sample ratio mismatch is, why it is checked first, and the worked chi-square.
• Sample size: why detecting a 2% lift on a 4% conversion rate needs about 940,000 users per arm.
• The honest weakness: client-side exposure loss that depends on the variant, and daily peeking.

How to read this board

1. Diagram, left to right: ASSIGN (a hash, not a lookup), LOG (exposure is the denominator), COMPUTE (join, then aggregate per user), DECIDE (is it real?). Press play for the walkthrough.
2. Below it, GUARDRAILS: the hourly path that kills a harmful experiment before the daily stats would.
3. Then the DATA MODEL: six tables, two of them with composite keys that ARE the definition of a unit.
4. Pages 1-3 · Requirements, Capacity and Storage: 3 billion events a day and 120 experiments live.
5. Page 4 · Availability - why assignment can never be down, and metrics may be a day late.
6. Page 5 · How it works, then Page 6 · Deployment, where the platform’s own changes are A/A tested.

The platform lets product teams define an experiment, split traffic between variants, and get an answer to "did B change metric X, by how much, and how sure are we?" It is two systems joined by a log: an assignment system in every request path, and an analysis system that runs on yesterday’s data.

Ask these before designing

• What is the unit? Logged-in user, device, session, or account? Bucketing by device splits one person across arms on phone and laptop; bucketing by account is right for B2B pricing.
• Can experiments overlap? Independent experiments use different salts and overlap freely. Experiments that touch the same surface go in a layer, share a salt, and take disjoint bucket ranges, so no user is in both.
• Where is the decision made - client or server? Server-side decisions can log exposure reliably at the decision point. Client-side ones depend on a phone flushing an event.
• How fast must a harmful experiment be stopped? Daily stats are for decisions. A crash-rate guardrail needs to be hourly.
• Will people peek? Yes. The question is whether the statistics are built for it (sequential tests) or pretend otherwise.

Functional

• Define experiments: variants, allocations, targeting, primary and guardrail metrics.
• Assign deterministically in every SDK, with no network call.
• Log first exposure per unit per experiment.
• Compute per-unit metrics from product events after exposure, for every running experiment, daily.
• Check sample ratio; compute effect sizes with confidence intervals and sequential bounds.
• Stop an experiment immediately by setting its allocation to zero.

Non-functional

• Assignment adds under 1 ms and has no failure mode - if config cannot load, every user gets control.
• Results by 09:00 for yesterday, for all running experiments.
• Guardrails within 1 hour.
• Reproducible: any result can be recomputed from the lake and metric definition version.

Out of scope

Feature flags without analysis, multi-armed bandits, causal inference on observational data.