Patterns · Scaling writes

Patterns

Scaling writes

How to accept more writes than one primary wants to handle, without pretending replicas will do it.

2 min read system-design

A primary serializes commits through one log. Read replicas do not change that. When write rate, write latency, or data size is the ceiling, you have a short menu.

Before you shard

Measure what a write costs. Every secondary index is a write. Every synchronous replica in another zone is latency. Drop indexes nobody queries. Batch many small inserts into one transaction if the producer can buffer for a few milliseconds. Move large payloads to object storage and write a pointer. These are not consolation prizes. They are often enough, and they make the later shard simpler.

Absorb bursts

If the average fits and the peak does not, put a durable log in front. The API validates, appends, returns 202, and consumers apply at a steady rate. This is the right shape for click ingestion and for any “we must not lose it, but the report can wait a minute” path. It is the wrong shape for “the user is staring at a seat map.” Size the consumer, alert on age, and apply backpressure when the log grows: 429 the least important producers first.

Shard

Partition by a key present on every write. Each shard has its own log and its own primary, so write throughput scales with shard count until a single key is the hotspot. A hot key needs its own treatment (suffix buckets, or a queue in front of that key only).

Place rows that must commit together on the same shard. Accept that a global uniqueness check is now a directed lookup: generate an id that hashes to the shard, or do the uniqueness insert on a shard chosen by the unique value.

Lose less on purpose

Synchronous quorum writes survive a node death and cost latency. Asynchronous writes are faster and can lose the tail. Pick per data type, not per cluster slogan. A metrics pipeline can drop. An outbox of payments cannot.

Say the number that forced your hand. “One primary does a few thousand non-trivial commits a second. This path is fifty thousand. I will shard by account id.” That sentence is the pattern.

Further reading

Check yourself

Answer the questions. Multiple choice is scored on this device only.

1. Which change increases write throughput on the primary?

2. You ack the client before the durable log has the write. You have chosen:

3. How do you keep idempotency working after you shard?