ZooKeeper and etcd

ZooKeeper and etcd run a consensus protocol (Zab, Raft) so a small set of nodes agree on a log of updates. Reads and writes of that log are linearizable as long as a quorum is alive. The data is a tree of small keys. Clients can watch a key and learn when it changes. Sessions expire if a client stops heartbeating.

That is a powerful and narrow tool. The cluster is typically three or five nodes. The dataset should fit in memory and stay modest. Throughput is nothing like Kafka or Redis. You use it for metadata that must not fork: who is the primary, what is the current config generation, which workers are alive, a lock that must have one holder.

Leases and locks

A lock is a key that exists only while the holder’s session is alive, or a lease with a TTL the holder refreshes. If you crash, you stop refreshing, and someone else may take it after the timeout. The timeout is the unavailability you accept in exchange for not waiting forever. Work done under the lock must be idempotent or fenced: a slow holder whose lease expired must not keep writing. A fencing token (a number that increases every time the lock is granted, checked by the storage system) is the grown-up version.

Watches

A watch is how workers learn that config changed without polling themselves into a stampede. When the watch fires, clients must jitter and re-read. A thundering herd after a leader election is a known way to turn a one-second failover into a one-minute outage.

What not to do

Do not store product rows, queues of user jobs, or anything large. Do not run a single-node “coordination service” and call it highly available; consensus needs a quorum, which means an odd number of nodes in more than one failure domain. And do not add etcd because the diagram looked empty. If the critical section is one table, Postgres can hold the lock. If the thing you are electing is the Postgres primary, the election store has to be outside Postgres, and then a coordinator earns its place.