Long-running tasks

Some work is slower than an HTTP timeout: transcoding, a report, a grading pass, a mail merge. The request’s only job is to record the work durably and hand back an id. Everything slow happens where a timeout cannot strand the user, and where a crash cannot lose the only copy of the intent.

The contract

POST /v1/jobs validates, writes a row, enqueues a message or relies on workers polling the table, and returns 202 with Location: /v1/jobs/{id}. The client polls that resource or listens for events. The body shows status, stage, and, when you are done, a pointer to the result. Do not return 200 for work you have not finished. Do not return 500 for work you have successfully queued.

The worker

A worker claims a job with a conditional update (status = queued and a lease). It heartbeats if the work is long. It writes progress on the row so the poll has something to say. On success it sets the output and status together. On a transient failure it increments an attempt counter and releases the lease, maybe with a delay. After N attempts it marks the job dead and stops. A human or a separate sweeper can requeue.

If the process is killed, the heartbeat stops, the lease expires, and another worker claims it. That second run must be safe. Store intermediate outputs under a key derived from the job id, not under a random temp path you will never find again.

What the queue is

A jobs table with SKIP LOCKED is a fine queue up to a surprising scale and has the virtue that the status and the queue are the same row. SQS or a Redis stream is fine when you want to remove polling load from the database. Kafka is fine when the “job” is really “every event in this stream.” Do not introduce a workflow engine for a single step. If the job grows branches, compensation, and human waits, move to the saga page.

sequenceDiagram
  participant C as Client
  participant A as API
  participant D as Jobs table
  participant W as Worker
  C->>A: POST /jobs
  A->>D: insert queued
  A-->>C: 202 job id
  W->>D: claim with lease
  W->>D: progress, then succeeded
  C->>A: GET /jobs/id