Your on-call is drowning in alerts — CPU spikes, a pod restarting, a queue depth crossing a threshold — most of which resolve themselves and don't correspond to anything users noticed. How would you redesign alerting around SLOs so the pages that fire are the ones worth waking someone for?

technical-conceptual · Senior level · software-engineering

What the interviewer is really asking

Assesses whether the candidate can shift alerting from low-level cause-based thresholds to symptom-based SLO alerting tied to user experience, understands error budgets and multi-window multi-burn-rate alerting to balance fast detection against noise, and can reason about why this reduces alert fatigue without missing real incidents.

What to say

• Diagnose the cause of the noise: those alerts are cause-based and infrastructure-level (CPU, restarts, queue depth), which don't map to user pain — a pod can restart with zero user impact — so you should alert on symptoms users actually feel, like error rate and latency against an SLO.
• Introduce SLOs and error budgets: define SLIs for the user-facing behavior, set a target, and page on how fast you're burning the error budget rather than on a raw instantaneous threshold, so transient blips that self-heal don't wake anyone.
• Use multi-window, multi-burn-rate alerting so a fast burn (large budget consumed in a short window) pages immediately while a slow burn opens a lower-urgency ticket, and demote the old cause-based signals to dashboards and ticket-level alerts for diagnosis, not paging.

What to avoid

• Just raising thresholds or adding longer delays to the existing cause-based alerts, which trades one noisy/blind setting for another and still pages on things users don't feel.
• Alerting on a single short window or a raw error-rate threshold, which either pages on every transient blip or misses slow, steady budget burns.
• Keeping every infrastructure metric as a page 'just in case,' which is exactly what causes the fatigue and trains on-call to ignore alerts.

Example answers

Strong: The noise comes from alerting on causes — CPU, restarts, queue depth — that don't map to user impact. I'd define SLIs for what users feel, like success rate and latency, set SLO targets, and alert on error-budget burn rate instead of raw thresholds, so a blip that self-heals never pages. I'd use multi-window multi-burn-rate alerts: a fast burn pages now, a slow burn just files a ticket. The old infra metrics stay as dashboards for diagnosis, not as pages.

Weak: I'd go through each noisy alert and bump the thresholds higher and add a five-minute delay so the self-resolving ones stop firing. That way we keep all the alerts we have but they're less sensitive, and on-call gets paged less often.