Expert task routing

What you are designing

A customer submits a batch of tasks: grade these answers, judge these pairs, review these documents. Each task needs an expert with a skill, sometimes two or three independent experts. You track quality and you pay for accepted work. Deadlines exist. Assume tens of thousands of open tasks and a few thousand experts, with a long tail of rare skills that are the actual bottleneck.

Requirements

• An expert asks for the next task they are qualified for and receives at most one claim.
• If they vanish, the task returns to the pool.
• Quality is measured, not assumed: gold tasks and agreement.
• A completed assignment pays once.
• A customer can see whether a batch will hit its deadline.

Correctness of the claim and of the payout matters more than shaving latency below a couple of hundred milliseconds.

Entities and API

Project, Task(status, skill, priority, lease_until, claimed_by), Assignment, GoldLabel, Payout.

POST /v1/tasks/next          -> claimed task or 204
POST /v1/assignments/{id}/heartbeat
POST /v1/assignments/{id}/submit
POST /v1/projects            batch create

/tasks/next is a claim, not a read. 200 returns the task and a lease deadline. 204 means the queue is empty for this skill, which is not an error and should not be retried in a tight loop. Heartbeat extends the lease. A heartbeat after expiry is 409 and the expert must claim again, because someone else may own the task. Submit is idempotent on the assignment id: a double submit stores one answer and returns the same receipt. Batch create returns 202 when the payload is stored asynchronously. Those tasks are not claimable until the object is durable.

High-level design

flowchart LR
  Expert --> API
  API --> DB[(Tasks in Postgres)]
  API --> Blob[(Payloads in object storage)]
  Sub[Submit] --> Qual[Agreement and gold checks]
  Qual --> DB
  Qual --> Pay[Payout ledger]
  Sweep[Lease sweeper] --> DB

Payloads (the long document, the model output) live in object storage. The task row is small and indexed for the claim query. A sweeper or the claim query itself ignores leases that have expired, setting status back to open.

Data model

CREATE TABLE tasks (
  id           BIGSERIAL PRIMARY KEY,
  project_id   BIGINT NOT NULL,
  status       TEXT NOT NULL,
  skill        TEXT NOT NULL,
  priority     INT NOT NULL,
  claimed_by   BIGINT,
  lease_until  TIMESTAMPTZ,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX tasks_claim ON tasks (skill, priority DESC, created_at)
  WHERE status = 'open';

WITH next AS (
  SELECT id FROM tasks
  WHERE status = 'open' AND skill = :skill
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE tasks t
SET status = 'claimed', claimed_by = :expert,
    lease_until = now() + interval '30 minutes'
FROM next
WHERE t.id = next.id
RETURNING t.*;

INSERT INTO payouts (assignment_id, expert_id, amount_cents)
VALUES (:a, :e, :c)
ON CONFLICT (assignment_id) DO NOTHING;

Deep dives

Who gets the hard task. The claim query can require a minimum quality score for the skill, stored on expert_skills and updated from gold results. Do not compute a fancy rank over every expert on the pull; filter in SQL, order by priority, and let the lease serialize. A project nearing its deadline bumps priority. Watch starvation: a low-priority project still needs a fairness term or it never runs.

Quality. Sprinkle gold tasks whose answer you know. Score experts on those, not only on speed. If gold tasks are recognizable, rotate them and do not reuse the same item until it is memorized. For normal tasks, two experts and an agreement threshold. Disagreement creates an adjudication task. An expert whose gold accuracy drops is paused, which is a status update, not a delete of their past labels (you may need to relabel those).

Deadlines. Throughput is active experts times tasks per hour, by skill. Compare that to remaining tasks and time left. Alert when the forecast misses, early enough to add experts or cut scope. This forecast is a query, not a new infrastructure.

Payout and cancel. The payout unique key is the assignment. A canceled project stops new claims and pays assignments already submitted. Do not try to claw back with an in-place delete; reverse with a ledger line if you must.

Levels

• Mid-level. Pull API, the claim SQL, a lease, payloads in object storage, idempotent submit.
• Senior. Partial index, gold and redundancy, payout uniqueness, deadline forecast, what happens on cancel.
• Staff. Skill scarcity, exploration of new experts, audit of who saw which customer data, and how a bad expert’s labels are retracted from a delivered dataset.