Like 0

Interview prep notes

jobsched.md Raw

System Design Interview: Designing a Distributed Job Scheduler

"Run this job every hour" or "run this once at 3 pm" for millions of jobs, with each run happening once and on time.

Requirements

  • Users register one-off and recurring (cron) jobs with a payload and a target (HTTP call or a queue message).
  • Jobs fire within a few seconds of their scheduled time.
  • Each run happens at least once, ideally exactly once; failures retry with backoff.

Data model

  • job(id, owner, schedule, payload, next_run_at, status)
  • job_run(job_id, scheduled_for, attempt, status, started_at, finished_at)

Finding due jobs

Index next_run_at. Scheduler nodes repeatedly claim a batch:

UPDATE job SET status = 'CLAIMED', claimed_by = :node, claimed_at = now()
WHERE id IN (
  SELECT id FROM job
  WHERE next_run_at <= now() + interval '5 seconds' AND status = 'SCHEDULED'
  ORDER BY next_run_at LIMIT 500
  FOR UPDATE SKIP LOCKED)
RETURNING id;

SKIP LOCKED lets many schedulers poll without fighting over the same rows. For very large scale, partition jobs by hash across shards and give each scheduler a set of shards.

A time-bucketed approach also works: put job IDs into a sorted set or a bucket per minute and pop the current bucket.

Executing

Claimed jobs are pushed to a work queue; stateless workers execute them and record the job_run. For a recurring job, compute the next next_run_at from the cron expression and reset it to SCHEDULED in the same transaction that records the run.

Reliability

  • If a scheduler dies after claiming, a reaper resets claims older than a timeout.
  • Workers may run a job twice after a timeout, so pass a run ID (job_id + scheduled_for) to the target as an idempotency key.
  • Retries with exponential backoff and a dead-letter state after N attempts.

Disclosure: I'm on the team behind Linkjob AI, an AI tool for job interviews.