An intermittent latency problem only shows up in production under load, spans several services, and you can't reproduce it. How would you use your observability tooling to find the root cause, and what does this incident tell you about gaps in that tooling?

system-design · Staff-principal level · software-engineering

What the interviewer is really asking

Probes whether the candidate can drive a real cross-service debugging session with telemetry — starting from SLO/metric symptoms, using distributed traces to localize the slow hop, drilling into logs, distinguishing high-percentile tail latency from the mean — and reflect on observability gaps (the difference between monitoring known unknowns and being able to ask new questions).

What to say

• Start from the symptom in metrics but look at the right statistic: an intermittent problem hides in the tail, so examine p99/p99.9 latency and error rate over time, not the mean (which a tail spike barely moves), and correlate the spikes with load, deploys, and dependency health to narrow the window.
• Use distributed traces to localize it: pull traces from the slow tail (this is exactly why you tail-sample to keep slow traces), compare a slow trace's span breakdown against a fast one, and find which hop or span is responsible — a downstream call, a lock, a GC pause, queue wait — then drill into that service's logs (joined by trace ID) for the specific detail.
• Reflect on the gap, which is the staff-level point: if you couldn't answer the question with existing telemetry, that's a signal your observability covers known failure modes but not new questions (low cardinality, no traces on the slow path, missing a key span or attribute). The fix is richer instrumentation and high-cardinality context so the NEXT unknown is answerable without a code change — observability vs mere monitoring.

What to avoid

• Look only at average latency and conclude 'it looks fine', missing that an intermittent tail problem barely moves the mean and lives in p99/p99.9.
• Try to reproduce it locally or add ad-hoc print logging and redeploy repeatedly, instead of using the production traces and telemetry that already capture the slow requests.
• Find and patch the immediate cause and stop, without asking why the existing observability couldn't answer the question and closing that gap so the next unknown is debuggable.

Example answers

Strong: First I'd look at the right statistic: an intermittent problem lives in the tail, so I'd examine p99/p99.9 latency and error rate over time — the mean barely moves on a tail spike — and correlate those spikes with load, recent deploys, and downstream health to bound the window. Then distributed traces localize it: I pull traces from the slow tail (which is exactly why tail-based sampling keeps slow traces), diff a slow trace's span breakdown against a fast one to find the responsible hop — a downstream call, lock contention, GC pause, queue wait — and drill into that service's logs joined by trace ID.

Weak: I'd add more logging around the slow area and deploy it, then watch the logs until I catch it happening and see what's slow.