API gateway - one front door for many services

Why this design

Almost every microservice architecture ends up with an API gateway, and interviewers ask about it to see whether you understand both halves of the trade: what moves into the gateway (TLS, authentication, rate limits, routing, logging) and what that centralisation costs (an extra hop on every request, and a component whose failure is everybody's failure). Real examples are Kong, AWS API Gateway, Apigee, Envoy as an edge proxy, and the NGINX config every company eventually grows into one.

What you will learn

• Which concerns belong in a gateway - and which (business logic, data joins) do not.
• How a gateway authenticates without calling the identity provider on every request: JWT signatures checked against cached public keys.
• Sizing: 50,000 requests/s at peak needs about 15 gateway instances and 50,000 counter writes/s for rate limiting.
• Why fail-open versus fail-closed on the rate limiter is a business decision, not a technical default.
• How to ship route changes to a fleet that every API depends on without breaking everything at once.

How to read this board

1. CALLERS on the left: the app user with a token, the partner with an API key.
2. GATEWAY PIPELINE across the top: edge (TLS and WAF), authenticate, rate limit, then the router hex.
3. GATEWAY STATE underneath: the identity provider's public keys, the red rate-limit counters, the route table and the control plane that publishes it.
4. SERVICES on the right: three of the twelve backends, and the access log stream.
5. Press play, read pages 1 to 6, then the DATA MODEL tables.
6. The conversation to have: name every way this one fleet can take down all twelve services, and for each one say what you would change.

Functional

• Expose all public APIs under one hostname, api.example.com, routed by path prefix (/v1/users, /v1/orders, /v1/catalog ...) to 12 backend services.
• Authenticate every request: end-user JWTs (from the company login) and partner API keys.
• Enforce rate limits per API key and per user according to a plan (free 60/min, pro 600/min, partner 6,000/min).
• Pass the verified identity to services in trusted headers (X-User-Id, X-Client-Id), stripping any the caller sent.
• Log every request with status, latency, caller and route.
• Let service teams add or change their routes without a gateway deploy.

Non-functional

• Added latency: p99 under 5 ms for the gateway itself.
• Availability 99.99 percent - at least as high as the best service behind it, since it is in front of all of them.
• A route change reaches all instances within 30 seconds and can be rolled back as fast.
• Peak 50,000 requests/s, scaling horizontally with no shared state on the request path except the rate counters.

Out of scope

• Response aggregation and business logic. A "backend for frontend" that stitches three calls into one is its own service behind the gateway, not part of it.
• Service-to-service traffic inside the network, which should not go back out through the front door.
• Issuing tokens - that is the login service; the gateway only verifies them.

Traffic

Average requests/s                 15,000
Peak requests/s                    50,000
Backend services                       12
Average request + response          6 KB

Gateway fleet

One instance (4 vCPU), comfortable   5,000 req/s
Peak / instance   50,000 / 5,000   =  10
Plus 50% headroom, 3 zones         =  15 instances

Where the gateway spends its time

TLS termination (on resumed sessions)       ~0.2 ms
JWT verify, RS256, key already cached       ~0.05 ms
Rate limit: one Redis round trip            ~0.5 ms
Route match + header rewrite                ~0.02 ms
Proxy to backend (connection pooled)        ~0.5 ms
                                            ---------
Gateway overhead                            ~1.3 ms typical

Most of the budget is the two network round trips - to the counter and to the backend. Verifying a JWT is cheap because it is pure CPU on a public key the gateway already holds; calling the identity provider to introspect each token instead would add a round trip and make the identity provider take 50,000 requests a second.

Rate limit counters

One INCR + EXPIRE per request    =  50,000 writes/s peak
One Redis primary, comfortable   ~ 100,000 ops/s
Utilisation at peak              ~  50 percent

Half of one Redis is fine today and a problem at twice the traffic. The fix is to shard counters by key across a Redis cluster, or to have each instance keep a local count and sync every 100 ms, accepting that the limit is approximate.

Bandwidth

50,000 x 6 KB  =  300 MB/s  (~2.4 Gbit/s) peak, through the fleet
Per instance   =  ~20 MB/s at 15 instances

A gateway holds little data of its own, and almost all of it is configuration that is read on every request and written rarely.

Configuration

Routes             ~300 rows x ~500 B     =  150 KB
API clients        50,000 x ~200 B        =   10 MB
API keys (hashed)  80,000 x ~150 B        =   12 MB
Plans              a handful

All of it fits in memory in every gateway instance. The database is the source of truth; instances load a snapshot at startup and receive changes by push, so a database outage stops config changes but not traffic.

Rate limit counters

Active callers in any minute     ~2,000,000
One counter per caller per window   ~60 B
Live counters                    =  ~120 MB

Counters expire with their window, so the store never grows; it only needs memory for one or two windows of active callers.

Access logs

Requests per day   15,000 x 86,400  =  1.3 billion
Per log line                          ~500 B
Per day                               =  ~650 GB
Compressed in object storage (~8:1)  =  ~80 GB/day
Keep 90 days                          =  ~7 TB

The gateway is everybody's dependency

If each of 12 services is 99.95 percent available, a client using all of them sees roughly their product. The gateway multiplies into every one of those: a 99.9 percent gateway caps every API at 99.9 percent. So the gateway fleet is run like the most critical service in the company - spread over three zones, autoscaled, with its own on-call.

What fails, and what it costs

• One gateway instance: the load balancer in front ejects it; the other 14 absorb its share.
• A bad gateway release: every API breaks at once. Canary it on 1 instance (about 7 percent of traffic) and watch error rates per route before going further.
• A bad route config: pushed everywhere in seconds. Validate before publish (upstream exists, no prefix shadows another), then roll out to a canary group first.
• Rate limit Redis (red): fail closed and a failover is a total outage; fail open and abusive clients go unlimited for its duration. The common answer is fail open with a local per-instance limit as a backstop, so nothing is completely unprotected.
• Identity provider: tokens are verified locally against cached keys, so an outage only stops NEW logins, not existing traffic - as long as the key cache is kept, not dropped, when a refresh fails.
• A slow backend: without per-route timeouts and connection limits, one slow service ties up gateway workers and starts failing everybody else's requests. Each route gets its own timeout and its own connection pool (a bulkhead).

What degrades instead of dying

• Control plane down: no route changes, traffic unaffected.
• Log stream backed up: logs buffer locally then drop; requests are never blocked on logging.
• Counters down: limits fall back to per-instance approximations.

One request, stage by stage

1. Edge: TLS is terminated and a web application firewall rejects obviously bad requests (oversized bodies, known attack patterns) before any work is done.
2. Authenticate: for a Bearer token, check the JWT signature with the identity provider's public key (JWKS, cached for 10 minutes), then the expiry and audience. For an API key, hash it and look it up in memory. Failure is 401. Success attaches the caller's id and plan.
3. Rate limit: increment the counter for (caller, current minute) in Redis and compare with the plan's limit. Over the limit is 429 with a Retry-After header.
4. Route: match the longest path prefix in the route table, strip identity headers the caller sent, add the verified ones, and proxy to that service over a pooled connection with the route's timeout.
5. Log: emit one access log line to the stream, without waiting.

Why verify tokens locally

A JWT is signed by the identity provider. Anyone with the public key can check the signature, so the gateway never needs to call the identity provider on the request path. The cost is revocation: a stolen token stays valid until it expires, which is why access tokens are kept short (5 to 15 minutes). The login-and-sessions board is about exactly that trade.

How routes reach the fleet

A service team edits its routes through the control plane. The control plane validates them, writes a new version of the route table and pushes it to every gateway instance, which swaps it in atomically between requests. Every instance reports which version it runs, so a rollout can be watched and a bad version rolled back by pushing the previous one.

What stays out

• Business rules: "free users cannot export" belongs in the service. The gateway knows plans for rate limits, not features.
• Joining data from several services: that is a separate aggregation service.
• Per-service special cases accumulated as plugins: each one is code every request runs, owned by a team that does not run the gateway.

Topology

• A network load balancer in front, spreading TCP across 15 gateway instances in 3 zones (5 per zone).
• Gateway instances are stateless: config in memory, counters in Redis, logs streamed out.
• Redis cluster for counters, with replicas in a second zone.
• Control plane and its Postgres in one zone - it is not on the request path.
• Backends are reached over the private network; they accept traffic only from the gateway's security group, so nobody can skip authentication by calling them directly.

Releasing the gateway

1. Deploy to 1 instance. Compare its 5xx rate, p99 latency and per-route error rates with the others for 15 minutes.
2. Then one zone (a third of traffic), then the rest.
3. Automatic rollback if 5xx rises above baseline by more than 0.5 percentage points.

Releasing a route change

1. Validate: upstream resolves, timeout set, no existing route shadowed.
2. Push to a canary group of 2 instances; watch that route's error rate.
3. Push to all. Keep the previous version one click away.

What to watch

Gateway-added latency (total minus upstream time), 5xx split into gateway-generated and upstream-generated, 401 and 429 rates per route, counter store latency, config version per instance, and open connections per upstream - a route whose connections climb is the slow service about to drag everybody down.