One of your services depends on a downstream that occasionally gets slow or fails. Walk me through how you'd keep that downstream's degradation from cascading into an outage of your own service — which resilience patterns you'd apply, and how they interact.

system-design · Staff-principal level · software-engineering

What the interviewer is really asking

Tests command of the core fault-isolation patterns — circuit breaker, bulkhead, timeouts, retry with backoff+jitter, graceful degradation — and crucially how they compose (retries inside the breaker, timeouts before bulkheads) rather than treating each as an independent checkbox.

What to say

• Start with timeouts and bulkheads to contain the blast radius: an aggressive per-call timeout so a slow downstream can't tie up your threads/connections indefinitely, and a bulkhead (a bounded, isolated connection/thread pool for that dependency) so its slowness can't exhaust the resources your other paths need — the failure stays in one compartment.
• Add a circuit breaker to stop hammering a failing downstream: closed → open when the error/latency rate crosses a threshold, half-open to probe recovery. Critically, retries live INSIDE the breaker — if the circuit is open there's nothing to retry — and retries use exponential backoff with jitter so your retries don't become a synchronized retry storm that keeps the downstream down.
• Design the fallback for when the dependency is unavailable: serve a cached/last-known value, a sensible default, or a degraded experience rather than propagating the error — so your service stays up at reduced fidelity. State the ordering explicitly: rate-limit before retry, retry inside the breaker, timeout before the bulkhead releases the resource.

What to avoid

• Add naive retries with no backoff, no jitter, and no circuit breaker, so a struggling downstream gets a retry storm that turns its slowness into a full outage (and yours).
• Rely on a circuit breaker alone with no timeout or bulkhead, so a downstream that's slow-but-not-erroring still ties up your resources and the breaker never trips.
• Have no fallback path, so the moment the dependency is unavailable your service returns errors to users instead of degrading gracefully.

Example answers

Strong: I'd contain the blast radius first: an aggressive per-call timeout so a slow downstream can't hold my threads indefinitely, and a bulkhead — an isolated, bounded connection/thread pool for that dependency — so its slowness can't starve my other code paths. The failure stays in its compartment instead of taking the whole service down.

Weak: I'd wrap the call in a retry loop so transient failures get retried a few times, and if it's still failing the request just errors out.