Load balancer - spreading traffic, and noticing who is dead

Why this design

A load balancer appears in nearly every system design answer, usually as one unexplained box. Interviewers ask about it directly to see whether you know what is inside: how a request is assigned to a server, how the balancer learns that a server is gone, what happens to the requests in flight when it does, and what happens when the balancer itself fails. In real systems this is an AWS ALB or NLB, an NGINX or HAProxy pair, Envoy, or a Kubernetes Service - the same decisions with different defaults.

What you will learn

• Layer 4 against layer 7: forwarding TCP connections versus reading HTTP requests, and what each can and cannot do.
• Round robin, least connections and consistent hashing - and why the choice matters less than people think.
• Health checks as numbers: interval x threshold = how long a dead server keeps receiving traffic.
• Shallow against deep checks, and how a deep check can take down a healthy pool.
• Connection draining, sticky sessions, and keeping the balancer itself from being the single point of failure.

How to read this board

1. FRONT DOOR on the left: the client, DNS, the active balancer hex, its standby, and the red health checker.
2. SERVER POOL in the middle: four app servers, one draining and one dead, plus the autoscaler that adds and removes them.
3. SHARED STATE on the right: the session store and the database every server depends on.
4. Press play for the walkthrough, then read pages 1 to 6, then the DATA MODEL under the pages.
5. The conversation to have is the red box. Pick the interval, the threshold and what the check tests, and say what each choice costs when a server dies and when the database blinks.

Functional

• Accept HTTPS on one public name (api.example.com) and terminate TLS.
• Spread requests across a pool of identical app servers.
• Detect unhealthy servers and stop sending them traffic; bring them back when they recover.
• Add and remove servers without dropping requests (deploys and autoscaling).
• Route by path or host when needed (/api to one pool, /static to another).
• Optionally keep a client on the same server (session affinity).

Non-functional

• Added latency under 1 ms at p99 for the balancer itself.
• Availability 99.99 percent for the front door - higher than any single server behind it.
• A dead server should stop receiving traffic within seconds, not minutes.
• No requests dropped during a planned removal (deploy, scale-in).
• Handle 20,000 requests/s and 200,000 concurrent keep-alive connections.

Out of scope

• Global traffic steering across regions (GeoDNS, anycast) - one level up from this board.
• Rate limiting and authentication, which belong to an API gateway.
• Service-to-service balancing inside a mesh, which is the same ideas run in every client.

Traffic

Average requests per second          6,000
Peak requests per second            20,000
Average response size                20 KB
Concurrent keep-alive connections  200,000
New TLS connections per second       2,000  (peak)

How many servers

One app server, comfortable          1,500 req/s
Peak / per server   20,000 / 1,500 = 13.3  ->  14
Plus headroom to lose a zone (N+2)       =  16 servers
At peak, each server takes   20,000 / 16 = 1,250 req/s

The board draws four servers to stay readable; the numbers assume sixteen spread over two zones. The extra two are not waste: they are what lets one server die, or one be drained for a deploy, without the rest going over their comfortable load.

Bandwidth through the balancer

Egress   20,000 x 20 KB        =  400 MB/s  (~3.2 Gbit/s) peak
         6,000 x 20 KB         =  120 MB/s average
Ingress  requests ~2 KB        =   40 MB/s peak

A layer 7 balancer has to decrypt and re-read all of this. 3.2 Gbit/s plus 2,000 TLS handshakes a second is within one modern balancer instance, but not with much room - which is why managed balancers scale out behind the scenes and why a self-run pair is sized for twice the peak.

The health check window

Interval 5 s  x  unhealthy threshold 3  =  up to 15 s
Requests sent to a dead server in that window
   1,250 req/s  x  15 s                   =  18,750
Interval 2 s  x  threshold 2              =   4 s  ->  5,000
Check traffic: 2 LBs x 16 targets / 5 s   =  6.4 checks/s

Health checks cost almost nothing, so the temptation is to make them very frequent. The limit is not load, it is flapping: a check every second with a threshold of one ejects a server for a single slow garbage-collection pause.

A load balancer stores very little, and that is a design goal: anything it keeps is something a standby has to have too.

In the balancer

Connection table   200,000 x ~300 B          =  ~60 MB
Config (pools, targets, rules, certs)        =  < 1 MB
Health state       16 targets x a few bytes  =  nothing

The connection table is the one piece of state that matters, and it is not replicated to the standby in this design. That is why a failover drops every open connection: the standby has never heard of them.

Behind the balancer

Active sessions            2,000,000
Per session (Redis)            ~1 KB
Session store             =   ~2 GB

Two gigabytes fits one Redis primary with a replica. Moving sessions out of server memory into this store is what makes sticky sessions optional: any server can serve any user, so removing one logs nobody out.

Access logs

Requests per day   6,000 x 86,400   =  518,000,000
Per log line                         ~400 B
Per day                              =  ~207 GB
Keep 30 days                         =  ~6.2 TB

The balancer's access log is the best single source of truth for latency and error rates, because it sees every request including the ones that never reached a server. Ship it to object storage, not to local disk.

A dead server

• Active checks: the balancer calls GET /healthz on every target every 5 s. Three failures in a row eject it; two passes bring it back.
• Passive checks (outlier detection): the balancer also watches real traffic. Five 5xx responses or connection resets in a row eject a target immediately, without waiting for the next active check. This closes most of the 15 second window.
• Retries: a request that fails to CONNECT can safely be retried on another server. A request that was sent and timed out can only be retried if it is idempotent (GET, or a POST with an idempotency key).

What the check should test

• Shallow (process answers): never ejects the pool by mistake, but misses a server that is up and broken.
• Deep (database reachable): catches the broken server, but when the database blips, every server fails together and the balancer ejects all of them.
• The usual answer: check what is local to the server (process, thread pool, disk, its own connection pool has free slots), not shared dependencies. And set a panic threshold: if more than half the pool looks unhealthy, ignore health and send to everyone, because the checker is more likely wrong than the pool.

A dead balancer

• Active/standby on a floating IP (VRRP / keepalived): the standby takes the IP within 1-3 s. Every established connection is lost because the connection table was not shared.
• Active/active behind DNS or anycast, or a managed balancer that spreads itself across zones: no single balancer is the front door. This is the better answer when you can have it.

A planned removal

Draining: the target is marked draining, receives no NEW requests, and finishes the ones it has for up to a drain timeout (30 s here). The deploy waits for drain before stopping the process. Skipping this is the most common reason deploys cause a small spike of errors.

Getting to the balancer

The client resolves api.example.com. DNS returns the balancer's public address - a floating IP held by whichever balancer is active. The client opens a TCP connection and a TLS session to that address; the balancer holds the certificate and terminates TLS.

Layer 4 or layer 7

• Layer 4 (TCP): picks a server when the connection opens and forwards bytes. Very fast, sees no HTTP, so it cannot route by path, retry a request or read a cookie. One long-lived connection carrying many requests all goes to one server.
• Layer 7 (HTTP): terminates the connection, reads each request, picks a server per request, and keeps its own pool of connections to the servers. Can route by path and host, add headers, retry, and spread a single client's requests. Costs more CPU per request.
• This board is layer 7.

Choosing a server

• Round robin: next server in turn. Fine when requests cost about the same.
• Least connections (drawn here): the server with the fewest requests in flight. Better when some requests are slow, because a server stuck on slow requests stops being picked.
• Power of two choices: pick two servers at random, send to the less loaded. Nearly as good as least connections and does not need an exact global count - the common choice for balancers that run as many instances.
• Consistent hashing on a key: the same user or cache key always lands on the same server. Needed when servers hold local state (caches); otherwise it only makes load uneven.

Sticky sessions

Affinity by cookie keeps a user on one server so in-memory session data stays valid. It also makes load uneven and turns every server removal into a mass logout. The better fix is the one drawn on the right: put sessions in a shared store and turn affinity off.

Adding and removing servers

The autoscaler watches CPU and request rate. When it adds a server, it registers it with the balancer as a new target; the target receives traffic only after it passes two health checks in a row. When it removes one, it deregisters, waits for drain, then terminates.

Topology

• Two balancer nodes in different zones (or one managed balancer that spans them). Health checks on, outlier detection on, panic threshold at 50 percent.
• Sixteen app servers across the same two zones, eight per zone, in one target group. Cross-zone balancing on, so losing a zone moves traffic rather than overloading the survivors in one zone.
• Redis for sessions with a replica in the other zone; affinity off.
• Access logs to object storage; metrics per target (requests, 5xx, latency, health state) to the monitoring system.

Rolling deploy through the balancer

1. Pick 2 of the 16 servers (12.5 percent). Mark them draining.
2. Wait up to 30 s for in-flight requests to finish.
3. Deploy, start, and wait for two passing health checks before they receive traffic again.
4. Watch their 5xx rate and latency against the rest for a few minutes, then continue in batches.

Changing the balancer itself

• Config changes (rules, weights, timeouts) are versioned and applied to the standby first, then failed over to, so the active node is never the one being edited.
• Certificate rotation: load the new certificate alongside the old one before expiry, never on the day.
• Any change to health check settings is tested by killing a server on purpose in staging and timing how long it keeps receiving traffic.

What to watch

Healthy target count per zone, 5xx from the balancer itself (no target available) versus from targets, per-target request share (one server taking far more than 1/16 means affinity or hashing is skewed), surge queue length, and TLS handshake rate.