Design a distributed job scheduler that can run millions of cron jobs reliably, at scale, across many worker nodes.

system-design · Senior level · software-engineering

What the interviewer is really asking

Tests understanding of distributed coordination, exactly-once execution, fault tolerance, and horizontal scalability of stateful systems.

What to say

• Job storage: persist jobs in a database (PostgreSQL or DynamoDB) with fields: job_id, cron_expression, next_run_at, status (pending/running/done), worker_id. Index on next_run_at for efficient polling.
• Scheduling: a scheduler process polls for jobs with next_run_at <= now and claims them atomically (UPDATE ... WHERE status='pending' AND next_run_at <= now RETURNING — row-level locking prevents double dispatch). Enqueue claimed jobs to a work queue (SQS/Kafka).
• Workers: pull from queue, execute job, update status to done/failed. Heartbeat mechanism: if a worker dies mid-job, a watchdog reclaims it after a timeout. Retry logic with max_attempts and backoff.

What to avoid

• Use a single cron daemon on one machine — single point of failure.
• Ignore exactly-once execution — the hardest part of a distributed scheduler.
• Poll the database for every job on every worker simultaneously without a claiming mechanism.

Example answers

Strong: The critical insight is atomic claiming: multiple scheduler instances must not dispatch the same job. Using 'SELECT FOR UPDATE SKIP LOCKED' in PostgreSQL, each scheduler atomically claims a batch of due jobs without contention — other workers skip the locked rows instead of blocking. This is how Postgres-backed queues like Rails Solid Queue, GoodJob, and Que avoid double-dispatch at scale; Redis-backed systems like Sidekiq achieve the same atomic claim differently, via an atomic BRPOP/LMOVE off a list.

Weak: Use cron on each server and just accept that some jobs might run twice.