Observability

Observability is how you answer “is it broken, and where?” without SSH. In a design interview you do not need a vendor tour. You need the three signals you would page on, and the id that ties a user’s failure to a log line.

Three kinds of telemetry

Metrics are numbers over time, cheap to keep, and the thing you alert on. For a request path, record rate, error rate, and a latency histogram (RED). For a worker, record queue age, success, and retry count. For a machine, CPU, memory, and disk (USE) explain a symptom. They are not the symptom.

Alert on p95 or p99 and on error ratio, ideally against an SLO (“99.9% of redirects under 50 ms”) with a burn-rate alert. A threshold on the mean hides the tail. A threshold on CPU pages you for a backup job.

Logs are discrete events. Give them a level, a request id, and structured fields. Do not log payloads that contain secrets, resumes, or payment data. Log the decision (“idempotency replay”, “upstream 503, attempt 2”) so a future you can reconstruct it.

Traces follow one request across services and through a queue. Propagate the trace id on the message. Without that, the worker’s failure is an orphan. You do not need tracing on day one of a design. You need to say the id survives the async boundary.

What “healthy” means

A process health check should fail when the process cannot do the job: the database pool is exhausted, or the worker cannot dequeue. A check that always returns 200 keeps a broken box in the load balancer.

For async systems the user-facing health is the lag. A website that returns 200 while grades sit for six hours is down, and only the queue-age alert knows it.

Cardinality and cost

Every unique combination of labels is a time series. Route and status code are fine. User id, raw URL, or exception message is how you knock over the monitoring cluster. High-cardinality facts belong in logs and traces, sampled if the volume is huge.

Say you would dashboard the deep-dive bottleneck specifically: cache hit rate, replica lag, 429s from the model provider, double-claim attempts. The metric should match the failure mode you already wrote down.