Payments & ledgers
Fraud detection - a decision that must fit in 120 milliseconds
Teaches how a risk decision is split between a synchronous call with a hard deadline and an asynchronous review that is allowed to be right.
What this board gets wrong on purpose
The tension
When the deadline is missed the scorer fails OPEN - it allows the payment - and that is the correct choice, because a risk engine able to block revenue will eventually block all of it. It is also the exact behaviour an attacker wants, and the attack is cheap: make the feature store slow and every decision becomes an allow. Nothing on this board distinguishes "the feature store is degraded" from "somebody is degrading the feature store", and the timeout rate is a dashboard rather than a circuit breaker. The second unresolved thing is feature freshness: velocity counters are written from a stream that is seconds to minutes behind at peak, so the six-card-attempts-in-one-minute pattern the rules exist to catch is invisible for exactly as long as the lag lasts - and the lag is worst precisely during the traffic spike an attack produces. Reading the counters synchronously from the write path would fix it and would also put a 40 ms dependency inside a 120 ms budget, so the design accepts staleness and does not measure what it costs.
1 · Requirements
This system answers one question - should this payment proceed - inside somebody else’s latency budget, and then answers it again more carefully when nobody is waiting.
Ask these before designing
- What does a false positive cost, in money? Until that number exists, no threshold can be chosen and every argument about the model is an argument about taste. Here: average basket 68 EUR, and about 40% of falsely declined customers never come back, so a false decline costs roughly 2.4x a basket in lifetime terms.
- Who owns the loss? If the merchant eats the chargeback, they will accept a looser threshold than the gateway would. If we eat it, we will not. The threshold is a commercial decision wearing an engineering costume.
- Is a "review" outcome usable at all? Holding a payment for a human is only possible if the product can say "we are checking" without shipping. A digital-goods merchant has no such state, and for them the three-way decision collapses to two.
- How fresh must features be? "Six attempts in the last minute" is a different system from "spend in the last 30 days". The first needs a synchronous counter; the second is a nightly batch. Most teams build one pipeline and are surprised that it serves neither well.
- What is the deadline, and what happens at it? Not "the model is fast". A hard number, and a stated default for when it is exceeded.
Functional
- Score a payment and return ALLOW, REVIEW or DECLINE within the deadline.
- Record what the model saw, not just what it said, for every decision.
- Queue REVIEW decisions for human analysts with the evidence attached.
- Accept chargeback labels weeks later and attach them to the original decision.
- Let a rule be added, shadowed and enabled without a deploy.
Non-functional
- p99 of 120 ms end to end, measured at the caller, not at the model server.
- Fail open at the deadline. Stated as a requirement rather than discovered as a behaviour. A risk engine that can block revenue will one day block all of it.
- Every decision is explainable. A stored feature vector and a rule trace, kept for the length of the dispute window. "The model said so" is not an answer to a regulator or to a merchant.
- 99.99% availability, with a defined answer when unavailable - which is ALLOW, and which is why the availability number matters less here than anywhere else on these boards.
- No PII in the feature store. Hashed identifiers only. The feature store is replicated to every region and is therefore the easiest thing here to leak.
Explicitly out of scope
Model training infrastructure, the payment rails themselves, chargeback representment workflow, identity verification and KYC. This board is about the shape of the decision, not about the mathematics inside it.
The rest of this board
Available on Tier Slate
This page publishes the question. The answer — 5 more written pages, an 8-step narrated walkthrough and a 5-table schema — is the board itself, and it opens in Tierslate.
- 5written pages
- 8walkthrough steps
- 5tables32 columns
5 pages behind this one
- Capacity estimation
- Storage estimation
- Availability
- How it works
- Deployment plan
More boards
Systems where losing a write is losing money.