RAG chatbot - the answer is only as good as the retrieval

Why this design

"Build a chatbot over our internal documents" is now the most common applied-AI system design question, and most answers spend their time on the model. The model is the part you cannot change and do not need to: given the right eight passages, any current frontier model writes a good answer. The design work is retrieval - how documents are cut into chunks, how a follow-up question becomes a searchable query, why vector search alone misses exact terms, how fifty candidates become eight - and trust: citations a reader can check, permissions that stop the bot quoting a document the asker cannot open, and an evaluation that measures retrieval separately from writing.

What you will learn

• Why chunk size (here 512 tokens with 64 overlap) is the most consequential parameter in the system, and why chunks carry their heading path.
• Why hybrid retrieval - dense vectors AND BM25, merged with reciprocal rank fusion - beats either alone on internal documents full of product codes and acronyms.
• Why a cross-encoder reranker turns 50 candidates into 8, and why it is affordable only because retrieval already cut ten million to fifty.
• How the prompt is assembled with numbered sources so the answer can cite [1]..[8], and what a grounding check can and cannot prove.
• Where the money goes (input tokens, not output), where the latency goes (time to first token), and why the 6,000-token prompt is the bill.
• Why permissions copied into the index go stale, and why an embedding-model upgrade is a full re-index.

How to read this board

1. Press Play and follow the online path left to right: THE QUESTION, RETRIEVE, GROUND.
2. Then the bottom row, INGEST AND EVALUATE: the offline pipeline that decided, weeks ago, what could be found at all.
3. 1 · Requirements, 2 · Capacity (tokens and latency), 3 · Storage (vectors), 4 · Availability, 5 · How it works, 6 · Deployment.
4. Finally the DATA MODEL, and the table that logs what was retrieved for every answer - the one that makes evaluation possible.

Ask these before drawing anything

• What are the sources, and how fast do they change? A wiki edited hourly, a drive of PDFs, a ticket system. Change rate sets the ingest design; "re-index everything nightly" works at 50,000 documents and not at 500,000.
• Who may see what? If the answer is "everyone sees everything", half this board disappears. It never is. Retrieval must filter by the asker's groups BEFORE ranking, never after generation.
• Must every answer cite? Yes for anything internal: a cited answer can be checked in five seconds, an uncited one cannot be trusted at all.
• What should happen when the documents do not contain the answer? "I could not find this in the documents" is a correct answer and must be a designed outcome, not a failure.
• Is it single-turn search or a conversation? A conversation means follow-ups like "and for contractors?", which are not searchable on their own and need rewriting.
• How will quality be measured? If there is no golden set of questions with known source passages, there is no way to tell whether a chunking change helped.

Functional

• Answer natural-language questions over 500,000 documents from four source systems, citing every claim with a numbered link to the passage it came from.
• Respect source permissions: never retrieve a chunk whose document the asker cannot open in the source system.
• Support follow-up questions within a conversation, using the last six turns.
• Reflect a document edit within 15 minutes and a deletion within 5.
• Log, for every answer, every chunk retrieved, its rank, whether it was cited, and the user's feedback.

Non-functional

• Time to first token p95 under 1.5 s; full answer p95 under 8 s. Streaming is not a nicety - it is what makes an 8-second answer feel fast.
• Retrieval recall@8 above 0.85 on the golden set, measured on every chunking, embedding or reranker change.
• Under 2 percent of answers containing a citation that does not support its sentence.
• Cost under 3 cents per answer at list prices.

Out of scope

• Agents that take actions (filing a ticket, sending an email). That is tool use with its own authorisation problem.
• Fine-tuning the generator on the corpus. It teaches style, not facts, and cannot cite.
• Structured data questions ("how many tickets last week") - those want text-to-SQL, a different board.