LLM inference serving - the GPU is the bill

Why this design

Calling a hosted model is an API problem. Serving your own - for cost at volume, for data that cannot leave, or for a fine-tuned open-weights model - is a hardware economics problem, and it is the deepest AI system design question there is. A GPU costs the same per hour idle or busy, its memory holds only so many conversations, and the two phases of generation stress it in opposite ways. Interviewers want to hear the arithmetic: how big the KV cache is per token, why decode is memory-bound, what continuous batching and paged attention buy, and what you would trade to get time-to-first-token down.

What you will learn

• How to size GPU memory: 70 GB of FP8 weights, about 206 GB of KV cache on four H100s, and 320 KB of KV per token - so about 640,000 tokens, or ~250 concurrent conversations.
• Why prefill is compute-bound and decode is bandwidth-bound, and what that means for TTFT (time to first token) and TPOT (time per output token).
• How continuous batching admits and retires sequences every step, and how paged KV blocks make that possible without fragmentation.
• How prefix caching, chunked prefill and speculative decoding each move the latency/throughput line - and when each one stops helping.
• How preemption works when the KV cache fills, how routing exploits the prefix cache, and why autoscaling GPUs is slow.
• How to price it: tokens per GPU-hour, and why utilisation sets the cost per million tokens.

How to read this board

1. Press Play: THE REQUEST, ROUTE AND ADMIT, then ONE REPLICA - prefill, the decode loop, the KV cache and what happens when it fills.
2. Then FLEET along the bottom: the metrics that drive scaling, and the minutes a new replica takes to exist.
3. 1 · Requirements, 2 · Capacity (the GPU arithmetic), 3 · Storage (memory), 4 · Availability, 5 · How it works, 6 · Deployment.
4. Then the DATA MODEL: every request's tokens, cached tokens, TTFT and TPOT, which is how a GPU bill becomes a per-tenant one.

Ask these before drawing anything

• Which model, and at what precision? A 70B dense model with grouped-query attention (80 layers, 8 KV heads of 128 dims), served in FP8. Precision halves memory and roughly doubles compute throughput on H100s; it must be validated against quality, not assumed.
• What are the latency targets, and which matters more? TTFT is what makes a chat feel responsive; TPOT is reading speed. Humans read at 5-10 tokens a second, so 25+ tokens a second per stream is "fast"; agents and batch jobs care about total throughput instead.
• What is the context-length distribution? Average 2,000 in and 400 out, but a long tail to 32,000. The tail decides KV capacity and the worst prefill stall.
• How much of each prompt is shared? Long system prompts and few-shot examples repeat across requests - here about 60 percent of input tokens. That is what prefix caching is worth.
• Interactive, batch, or both on one fleet? Both, here, with priority at admission - the board's tension.
• What GPUs can you actually get, and how fast? Capacity is often reserved months ahead; autoscaling only works inside what was reserved.

Functional

• OpenAI-compatible chat completions with streaming (server-sent events), stop sequences, max_tokens and temperature.
• Per-tenant limits on concurrent sequences and tokens per minute, with interactive above batch priority.
• Report prompt, cached-prompt and output tokens per request for metering.
• Cancel generation when the client disconnects, freeing its KV blocks within one step.

Non-functional

• Interactive: TTFT p95 under 800 ms for prompts under 4,000 tokens; TPOT p95 under 40 ms (25+ tokens/s per stream).
• Throughput: at least 8,000 output tokens/s per four-GPU replica at the interactive latency targets.
• Fleet utilisation above 60 percent of GPU-hours over a day - below that, a hosted API is usually cheaper.
• No request silently truncated: a sequence preempted for memory resumes or fails explicitly.

Out of scope

• Training and fine-tuning (another cluster, another scheduler).
• The model's quality, safety filtering and prompt design.
• Multi-region active-active - this fleet is one region; the gateway in front fails over to a hosted provider.