API gateways and load balancers

These two boxes sit in front of your services and people mix them up. Split them by what they are allowed to understand.

A load balancer has a list of targets and a health check. It sends new connections to targets that are healthy, using round-robin, least-connections, or a hash of the client. Layer 4 does this on TCP. Layer 7 terminates TLS and can route api.example.com/v1/search to a different fleet from /v1/upload. It does not know what an order is.

An API gateway is an L7 edge with opinions: it checks the token, applies rate limits, requests a size cap, rewrites a path, and emits a standard error. It might fan in several public APIs. It should not contain product rules and it should not be the only copy of authorization. Services still check that this caller can touch this id, because internal callers exist.

Health and deploys

A health check that always returns 200 is decoration. A health check that requires every dependency including the data warehouse will drain the fleet during a partial outage and turn a brownout into a blackout. Check that the process can serve the request it is about to receive: thread pool not exhausted, local cache loaded, primary database reachable for a write tier.

On deploy, mark the target draining, wait out the p99 of in-flight work (and the websocket you decided was sticky to that process), then stop it. New tasks start receiving traffic only after they pass a readiness check, ideally with a slow start so a cold cache does not take a full share immediately.

Stickiness and streams

WebSockets and long polls want either a shared pub/sub (any node can finish the delivery) or stickiness. Stickiness fails when that node dies, so the client must reconnect and resume from a cursor anyway. Design the resume path. Do not design a world where one process is precious.

Internal calls

Once traffic is inside the mesh, a header the gateway set (“this user id”) is only as trustworthy as the network. Sign it or use mTLS so a random pod cannot invent a user. Timeouts and retries at this layer follow the idempotency rules: retry GETs, retry POSTs only with a key, and keep a budget so five services do not multiply a single user click into a retry storm.