Your dashboards are full of green graphs but you still find out about outages from customers before your monitoring tells you. How do you redesign your alerting so it catches what actually matters?
technical-conceptual · Senior level · cloud-devops-security
What the interviewer is really asking
Probes whether the candidate can shift from cause-based, resource-utilization alerting to symptom-based alerting on user-facing signals, understands why CPU/memory alerts miss real outages, and can reason about actionability so every page is something a human must act on — rather than adding more dashboards.
What to say
- Diagnose the real gap: green CPU/memory graphs say the boxes are healthy, but they don't tell you whether users can complete requests — you're alerting on causes, not on the symptom the customer feels.
- Alert on user-facing symptoms: pages should fire on the things users experience — error rate, latency, success rate of key journeys — typically expressed as SLO/SLI burn, so the alert tracks pain, not resource noise.
- Make every page actionable: if a human can't or needn't act on it, it's a ticket or a dashboard, not a page — ruthless actionability is what kills both the false-negative (missed outage) and the alert fatigue that hides it.
What to avoid
- Adding more resource-utilization alerts (CPU, memory, disk) and assuming more thresholds means better coverage — those miss the user-facing failures entirely.
- Paging on every anomaly so 'nothing slips through,' which buries the real signal in noise and trains responders to ignore the pager.
- Treating the fix as buying a fancier dashboard rather than changing what you alert on and why.
Example answers
Strong: Green CPU graphs only prove the hosts are up, not that users can check out — I was alerting on causes, not symptoms. So I'd flip it: page on the symptoms users actually feel, like the error rate and latency of the key journeys, expressed as SLO burn. Then I'd ruthlessly enforce actionability — if a human can't do anything about an alert, it becomes a ticket or a dashboard panel, not a page. That closes the gap and cuts the noise that was hiding the real signal in the first place.
Weak: I'd add more monitoring — alerts on CPU, memory, disk, and network for every service — and build a big overview dashboard so the on-call can watch everything in one place and catch problems as soon as a metric looks off.