Queues and streams

A queue lets the producer finish without waiting for the slow step. That is the entire product reason. Everything else is about not losing, reordering, or infinitely buffering the work.

Two different tools

A work queue (SQS, a Redis list, a Postgres table claimed with FOR UPDATE SKIP LOCKED) hands each job to one worker. If the worker dies, the job becomes visible again after a visibility timeout or a lease. You get competing consumers, delays, and a dead-letter queue.

A log (Kafka, Kinesis, Pulsar) appends records to a partition and keeps them. Consumers track an offset. Many consumer groups each see every record. Ordering holds inside a partition, so you key by the entity that must not be reordered (the account, the conversation). You can replay yesterday. You cannot easily say “give this one record to one of fifty workers” without extra machinery, because a partition has one active consumer inside a group.

Use the log when more than one system wants the same facts, or you want replay. Use the work queue when you want a job done once.

Delivery, said honestly

At-most-once: ack first, then work. You can lose jobs. At-least-once: work, then ack. You can see a job twice. Exactly-once is a property of the outcome, not of the queue. You get it by making the consumer idempotent: a unique key, a transaction that writes the offset and the result together, or a dedup table. Say “at-least-once plus idempotent consumer.” Do not say “Kafka is exactly-once” and stop.

Operational basics

Bound the queue. Alert on age of the oldest message, not only on depth. A deep queue of fresh work is scale. A short queue of hour-old work is a stuck consumer.

Retries need a cap and a dead-letter destination. A poison message that always throws will stall a Kafka partition if you refuse to move past it. Move it aside and page a human.

Delay and priority are easy in a purpose-built queue and awkward in a plain log. If the product is “run this in an hour” or “premium customers first,” say so when you pick the tool.

flowchart LR
  API[API] -->|commit| DB[(DB + outbox)]
  API -->|202| User
  DB --> Q[Queue or log]
  Q --> W[Worker]
  W --> DB