Design a notification system that sends order-status updates to users across email, SMS, and push. How do you keep it reliable and decoupled from the services that trigger notifications?

system-design · Mid level · software-engineering

What the interviewer is really asking

Assesses understanding of asynchronous, queue-based decoupling, third-party delivery failure handling, and at-least-once delivery semantics — the realistic concerns of a fan-out notification pipeline.

What to say

• Decouple producers from delivery with a queue or event bus: the order service publishes an event and returns immediately; a notification service consumes it, so a slow email provider never blocks order processing.
• Handle the multi-channel fan-out and per-channel failure: a worker per channel calls the provider, retries transient failures with backoff, and routes messages that keep failing to a dead-letter queue for inspection instead of losing or infinitely retrying them.
• Address delivery semantics and abuse: queues give at-least-once delivery so consumers should dedupe (idempotent sends keyed by event id), respect user channel preferences and quiet hours, and rate-limit per recipient to avoid spamming.

What to avoid

• Calling the email/SMS provider synchronously inside the order-placement request, coupling order latency and success to a third party.
• Assuming exactly-once delivery from the queue and skipping any dedupe, so a redelivered message double-notifies the user.
• Having no path for permanently failing messages — no retries with backoff and no dead-letter queue — so failures either vanish or loop forever.

Example answers

Strong: The order service publishes an order_status_changed event to a topic and is done — it doesn't know or care how notifications get sent. A notification service subscribes, looks up the user's enabled channels and preferences, and enqueues per-channel jobs. Each channel worker calls its provider with retry-and-backoff; messages that exhaust retries go to a DLQ so we can alarm and replay after fixing the issue. Because the queue is at-least-once, each send is idempotent keyed by event id plus channel, so a redelivery doesn't double-text the customer.

Weak: When an order ships, the order service loops over the user's channels and calls the email and SMS APIs right there before returning the response.