Content moderation - deciding before everyone sees it

Why this design

Every product that lets strangers post - comments, reviews, marketplace listings, chat, profile photos - needs moderation, and almost every first design gets it wrong the same way: it treats moderation as one function call that returns allowed or removed. In practice there are two very different deciders. A machine answers in a couple of hundred milliseconds and is confidently wrong at the edges; a person takes a minute, is expensive, and there are never enough of them. The design question is how to split the work between the two and, above all, what the post looks like to everyone while the slow one has not answered yet.

What you will learn

• Why known-bad media is caught by perceptual hash matching before any classifier runs, and why that one lookup is the cheapest, most accurate check on the board.
• How two thresholds on a classifier score split traffic into allow, limit and remove, and how moving either one by 0.05 changes reviewer headcount by hundreds.
• The LIMITED visibility state: visible to the author and followers, excluded from ranking, search and recommendations until a person decides.
• Why the review queue is ordered by expected harm (severity times projected reach), not by arrival time.
• How reviewer verdicts become labelled data, and the feedback loop that can make the classifier drift towards whatever the reviewers already do.
• Appeals as a second pass by a different reviewer, and why a decision is an append-only log row rather than a flag on the post.

How to read this board

1. Press Play. The walkthrough follows one post: SUBMIT on the top left, through the FAST GATE, into VISIBILITY on the right.
2. Then the SLOW QUEUE along the bottom - reports, reviewers, decisions, appeals - and the dashed wire back to the gate that retrains it.
3. 1 · Requirements - what must be decided synchronously and what may wait.
4. 2 · Capacity and 3 · Storage - why the classifier fleet is the easy part and the reviewer count is the hard one.
5. 4 · Availability - what the gate does when a detector is down.
6. 5 · How it works, then 6 · Deployment plan, then the DATA MODEL at the bottom.

Ask these before drawing anything

• Which harms are in scope, and are any of them illegal everywhere? Child sexual abuse material and terrorist content have legal reporting duties and hard removal deadlines; spam and rudeness do not. Different policies get different thresholds, queues and retention.
• Must a post be checked before anyone sees it, or may it publish and be cleaned up? Pre-moderation is safe and slow; post-moderation is fast and leaves screenshots. The answer is usually different per surface.
• How long may the author wait on the submit button? Past about one second people resubmit, and you moderate the same post twice.
• Is there a legal clock? The EU Digital Services Act requires a statement of reasons for every restriction and a working appeal route; Germany's NetzDG gave 24 hours for manifestly illegal content.
• What does the author see when their post is held or removed, and can they appeal?

Functional

• Every new post and media item passes an automatic gate before it is distributed.
• The gate returns one of four outcomes: allow, limit, remove, or remove-and-report (for the categories with legal reporting duties).
• Uncertain posts and user-reported posts enter a human review queue ordered by expected harm.
• Every restriction produces a decision row with the policy, the evidence and who or what decided, and a notice to the author.
• Authors can appeal; an appeal is decided by a different reviewer than the original.

Non-functional

• Gate latency 300 ms at p99, including media hashing, for text and images. Video is checked on the first frames synchronously and fully in the background.
• Known illegal media matched against the hash list: recall above 99 percent, zero tolerance for being distributed.
• Grey-zone posts reviewed within 4 hours at p90 on a normal day; reports on the highest severity class within 30 minutes.
• The gate fails safe per policy - never silently allow everything because one model is down.

Out of scope

• Writing the policies themselves. The system enforces a policy table; deciding what is harmful is a trust and safety team's job.
• Account-level abuse (fake accounts, spam rings). It feeds this system a signal, it is not this system.