System Design Interview: Designing a Distributed Job Scheduler
"Run this job every hour" or "run this once at 3 pm" for millions of jobs, with each run happening once and on time.
Requirements
- Users register one-off and recurring (cron) jobs with a payload and a target (HTTP call or a queue message).
- Jobs fire within a few seconds of their scheduled time.
- Each run happens at least once, ideally exactly once; failures retry with backoff.
Data model
job(id, owner, schedule, payload, next_run_at, status)job_run(job_id, scheduled_for, attempt, status, started_at, finished_at)
Finding due jobs
Index next_run_at. Scheduler nodes repeatedly claim a batch:
UPDATE job SET status = 'CLAIMED', claimed_by = :node, claimed_at = now()
WHERE id IN (
SELECT id FROM job
WHERE next_run_at <= now() + interval '5 seconds' AND status = 'SCHEDULED'
ORDER BY next_run_at LIMIT 500
FOR UPDATE SKIP LOCKED)
RETURNING id;
SKIP LOCKED lets many schedulers poll without fighting over the same rows. For very large scale, partition jobs by hash across shards and give each scheduler a set of shards.
A time-bucketed approach also works: put job IDs into a sorted set or a bucket per minute and pop the current bucket.
Executing
Claimed jobs are pushed to a work queue; stateless workers execute them and record the job_run. For a recurring job, compute the next next_run_at from the cron expression and reset it to SCHEDULED in the same transaction that records the run.
Reliability
- If a scheduler dies after claiming, a reaper resets claims older than a timeout.
- Workers may run a job twice after a timeout, so pass a run ID (job_id + scheduled_for) to the target as an idempotency key.
- Retries with exponential backoff and a dead-letter state after N attempts.
Disclosure: I'm on the team behind Linkjob AI, an AI tool for job interviews.