Recommendation engine - retrieve a thousand, rank twenty

Ask these before drawing anything

• What is being optimised, in one sentence, and who agreed to it? Clicks, watch time, purchases and long-term retention give four different systems, and optimising the easy one usually damages the important one.
• How big is the catalogue and how fast does it change? Ten thousand items that change monthly needs none of this. Ten million that change hourly needs all of it, and the difference is entirely in candidate generation.
• How fresh must a recommendation be? If a purchase must change the next page, that is a session-level requirement and needs real-time features. If tomorrow is fine, the whole right-hand side of this board can be a nightly batch and cost a tenth as much.
• What must NEVER be recommended? Age-restricted, region-locked, out of stock, already bought, blocked by the viewer. These are correctness rules and they belong before the model, not inside it.
• Is there an editorial or commercial override? There always is eventually, and building it in from the start is much cheaper than retrofitting a rules engine around a model.
• How will this be evaluated? If the answer is offline AUC only, the system will ship regressions confidently. If there is no experiment framework, there is no recommendation system - only a model.

Functional

• Return 30 ranked items for any viewer within 100 ms at p95, including a viewer seen for the first time.
• Retrieve candidates from at least three independent sources so that the failure of one degrades quality rather than emptying the feed.
• Exclude what the viewer has already seen in this session and what they have interacted with recently, unless the item type says otherwise.
• Log every impression with its position, the model version that produced it and the feature values used - without this the training data is unusable and every experiment is unattributable.
• Retrain nightly and promote a model only after an offline evaluation and a shadow run on real traffic.
• Support holding a percentage of traffic on the previous model, sticky per viewer, for as long as an experiment runs.

Non-functional

• p95 end-to-end 100 ms, p99 150 ms. Past that the app shows a fallback, because a slow feed is worse than a generic one.
• The feed must never be empty. Every failure mode degrades to trending, and trending is served from a cache that is refreshed independently of everything else on this board.
• Feature skew between offline and online computation must be measured continuously and kept under 1 percent per feature. Unmeasured skew is the default state and it silently invalidates every offline result.
• Model rollback must take under one minute and must not require a deploy.
• No viewer-level data crosses a region boundary on the serving path.

Out of scope

• Search. A query is an explicit intent and deserves a different retrieval path, even though it shares this index.
• Ads. The auction has a different objective and a hard budget constraint, and mixing the two objectives into one score is how both get worse.
• Explanations. Worth having, and honest ones are much harder than they look once the ranker is a gradient-boosted ensemble of four hundred features.