LLM gateway - one API in front of every model provider

Why this design

Every company with more than a handful of LLM features ends up building this, usually after the first surprise invoice. Forty services each holding their own provider keys means nobody can answer "what did the support bot cost last week", one runaway batch job can exhaust the organisation's tokens-per-minute limit and take every other feature down with it, and an outage at one provider is an outage everywhere. The gateway puts one API in front of all of it. Interviewers like it because it looks like an ordinary API gateway and is not: the unit is the token, cost is only known after the response, and failover changes the product.

What you will learn

• Why rate limits and budgets are in tokens per minute and dollars, not requests, and how a reservation is placed before a call and settled after it.
• Why the gateway meters from the provider's reported usage rather than its own tokenizer, and stores cost in integer micro-dollars with the price snapshotted.
• How routing works: model aliases, a primary and a fallback per alias, circuit breakers per provider and model, and why failover is a behaviour change.
• What caching can do here: exact-match response caching (rarely hits for chat), and provider-side prompt caching of long shared prefixes (hits constantly, and is lost on failover).
• Why the gateway must add under 10 ms and never buffer a stream.

How to read this board

1. Press Play and follow a request: CALLERS, THE GATE, ROUTE - and back out as a stream.
2. Then METER along the bottom: usage events, the priced ledger, the per-team chargeback, and the dashed wire that settles a reservation.
3. 1 · Requirements, 2 · Capacity (the token arithmetic), 3 · Storage, 4 · Availability, 5 · How it works, 6 · Deployment.
4. Then the DATA MODEL, where cost is an integer and the price is copied onto every usage row.

Ask these before drawing anything

• How many providers, and are any models self-hosted? Two hosted providers plus a self-hosted open-weights model for cheap bulk work, here. Each has a different API, tokenizer, price and rate limit.
• Is the API one shape for all of them? Yes - one chat-completions-style request and streaming response, translated per provider. Callers ask for an ALIAS ("fast", "smart", "embed"), never a vendor model string, so routing can change without a deploy anywhere else.
• Who pays? Every request carries a virtual key that belongs to a team; every token is charged back to that team monthly.
• Hard or soft budgets? Soft at 80 percent (alert), hard at 100 percent (reject) for most teams; production-critical keys get an overdraft.
• May prompts be logged? Not by default. Prompts contain customer data; the gateway logs token counts and metadata, and full bodies only for keys that opt in, redacted, for 7 days.
• Is failover allowed to change the model? It has to be, since no two providers serve the same model. Each alias declares which fallbacks are acceptable.

Functional

• One OpenAI-compatible endpoint for chat, embeddings and batch, translated to each provider.
• Authenticate virtual keys; map key to team, allowed aliases and limits.
• Enforce per-key and per-team tokens-per-minute and monthly spend limits.
• Route each alias to a primary model with ordered fallbacks; fail over on 429, 5xx, connection errors and time-to-first-token timeouts.
• Meter input, cached input and output tokens per request from the provider's usage report, price them, and attribute the cost to a team within a minute.
• Optional exact-match response cache, opt-in per key, for deterministic (temperature 0) workloads.

Non-functional

• Added latency under 10 ms p99 before the first byte goes to the provider; zero added latency per streamed token.
• Availability 99.95 percent for the gateway itself - higher than any single provider, which is the point of routing.
• Metering loss under 0.01 percent of tokens, reconciled against provider invoices monthly within 1 percent.

Out of scope

• Prompt management and versioning (a different tool upstream).
• Evaluation of output quality - the gateway knows the cost of an answer, not whether it was good.
• Agent orchestration. Tool calls pass through as ordinary messages.