Job scheduler

What you are designing

Users or services register jobs: send this email at 9:00, recompute this report every hour, retry this webhook in five minutes. A fleet of workers runs them. Assume millions of scheduled jobs, a burst at round clock times, and handlers that can crash mid-way. You are designing the scheduler, not every handler.

Requirements

• Register a one-shot or a recurring job.
• Run it close to the due time (define the slack, for example within 10 seconds, not within a millisecond).
• Survive worker and scheduler process death.
• Avoid two workers doing the same logical run, or make that harmless.

A job that runs twice must not send two emails. That is the handler’s idempotency, and the scheduler must give the handler a stable run id to key on.

Entities and API

Job(id, owner, schedule, next_run_at, payload_ref), Run(job_id, run_id, state, lease_until).

POST /v1/jobs          { schedule | run_at, payload }
DELETE /v1/jobs/{id}
GET  /v1/jobs/{id}/runs

Create returns 201 and a job id. The same idempotency key returns that job instead of a second schedule. Delete returns 204 and stops future fires. A run that is already claimed either finishes or is cancelled, and you say which before you write the code. GET .../runs is the audit trail, paged by time, not a scan of every payload. A run id is stable for one scheduled fire, so a worker that crashes and retries does not send the payload twice.

High-level design

flowchart LR
  API --> DB[(Jobs)]
  W1[Worker] -->|claim due| DB
  W2[Worker] -->|claim due| DB
  W1 --> H[Handler]
  H --> DB

There is no special leader if workers can claim safely. A optional dispatcher can push due job ids onto a queue so the database is not polled by thousands of workers. Either way the claim is conditional. A crashed worker’s lease expires and the run becomes claimable again under the same run id.

Data model

CREATE TABLE jobs (
  id           BIGSERIAL PRIMARY KEY,
  owner_id     BIGINT NOT NULL,
  next_run_at  TIMESTAMPTZ NOT NULL,
  schedule     TEXT,
  payload_ref  TEXT NOT NULL,
  enabled      BOOLEAN NOT NULL DEFAULT true
);
CREATE INDEX jobs_due ON jobs (next_run_at) WHERE enabled;
CREATE TABLE runs (
  job_id      BIGINT NOT NULL,
  run_at      TIMESTAMPTZ NOT NULL,
  state       TEXT NOT NULL,
  lease_until TIMESTAMPTZ,
  attempts    INT NOT NULL DEFAULT 0,
  PRIMARY KEY (job_id, run_at)
);

The primary key of runs is the idempotency key the handler uses (“email for job 9 at 09:00”).

Deep dives

Claim. Same shape as the task queue on the Postgres page: select due jobs FOR UPDATE SKIP LOCKED, insert or update the run to leased, commit. The handler executes, then you set the run succeeded and advance next_run_at in one transaction. If you advance the schedule before the handler finishes, a crash skips or overlaps. If you advance only after success, a poison job blocks the schedule unless you count attempts and park it.

Time. Store UTC. Recurrence is “next instant after this one in this timezone,” and daylight-saving time will duplicate or skip an hour. Say you use a library and you test the DST boundary. Do not write your own cron parser in the interview; say what it must return.

Scale and isolation. Shard jobs by owner so a poll touches an index of what is due, not a scan. Cap concurrent runs per owner. Jitter anything scheduled at :00. If the due set is huge, a dispatcher pulls a page and enqueues ids so workers do not all hammer the same index.

Failure. Attempts, backoff, and a dead state the owner can see. Alert on age of the oldest due job. That metric is the scheduler’s health, more than CPU.

Levels

• Mid-level. Durable jobs, a due index, a conditional claim, a run record.
• Senior. Lease expiry, idempotent run id, recurrence advanced in the right transaction, noisy-neighbor cap.
• Staff. DST and clock assumptions, shard or partition plan, push versus poll, and the SLO on lateness versus the cost of polling.