Kafka

Kafka is a distributed log. Producers append records to a topic. The topic is split into partitions. Each partition is an ordered, durable sequence replicated across brokers. Consumers read at their own pace and remember an offset.

That design is excellent at fan-out of facts: one write, many independent readers, replay if a reader had a bug. It is a clumsy work queue. Know which one you are drawing.

Partitions and order

Order is guaranteed inside a partition, not across them. Choose the key to be the entity whose events must stay ordered. All messages for one conversation share a partition. The cost is skew: a hot key is a hot partition, and you cannot parallelize that key without breaking order. If you need both, partition by key for the ordered stage and fan out later.

More partitions means more parallelism and more files, leaders, and rebalance pain. Pick a number you can live with. You can add partitions later, and it changes key placement, so it is not free.

Consumer groups

A group is a set of processes sharing the work of one subscription. Partitions are assigned among them. A second group sees every record again. That is how billing and notifications both observe orders without coordinating.

Rebalance happens when members join, leave, or die. A slow handler that exceeds the poll interval looks dead and causes a rebalance storm. Bound the handler or the batch.

Delivery

The default mental model is at-least-once: you process, then commit the offset. A crash in between redelivers. Idempotent consumers make that safe. Kafka’s own transactions can make “consume and produce to another topic” atomic. The moment the effect is an email, a charge, or a row in Postgres, you are back to an idempotency key.

Retention keeps data for a configured time or size. Compaction keeps the latest record per key. Click streams want retention. A changelog of account state wants compaction. Say which.

Producers should use acks that wait for replicas if the data matters, and they should retry with idempotent producer settings so a retried append does not duplicate inside the log. That still does not dedupe your consumer’s side effect.