# System Design Interview: Designing a Distributed Job Scheduler "Run this job every hour" or "run this once at 3 pm" for millions of jobs, with each run happening once and on time. ## Requirements - Users register one-off and recurring (cron) jobs with a payload and a target (HTTP call or a queue message). - Jobs fire within a few seconds of their scheduled time. - Each run happens at least once, ideally exactly once; failures retry with backoff. ## Data model - `job(id, owner, schedule, payload, next_run_at, status)` - `job_run(job_id, scheduled_for, attempt, status, started_at, finished_at)` ## Finding due jobs Index `next_run_at`. Scheduler nodes repeatedly claim a batch: ```sql UPDATE job SET status = 'CLAIMED', claimed_by = :node, claimed_at = now() WHERE id IN ( SELECT id FROM job WHERE next_run_at <= now() + interval '5 seconds' AND status = 'SCHEDULED' ORDER BY next_run_at LIMIT 500 FOR UPDATE SKIP LOCKED) RETURNING id; ``` `SKIP LOCKED` lets many schedulers poll without fighting over the same rows. For very large scale, partition jobs by hash across shards and give each scheduler a set of shards. A time-bucketed approach also works: put job IDs into a sorted set or a bucket per minute and pop the current bucket. ## Executing Claimed jobs are pushed to a **work queue**; stateless workers execute them and record the `job_run`. For a recurring job, compute the next `next_run_at` from the cron expression and reset it to SCHEDULED in the same transaction that records the run. ## Reliability - If a scheduler dies after claiming, a reaper resets claims older than a timeout. - Workers may run a job twice after a timeout, so pass a run ID (job_id + scheduled_for) to the target as an idempotency key. - Retries with exponential backoff and a dead-letter state after N attempts. --- *Disclosure: I'm on the team behind [Linkjob AI](https://www.linkjob.ai/), an AI tool for job interviews.*