Deduplication

Dedup means two things, and designs fall over when they use the tool for the wrong one.

Exact

The same logical event arrives twice: a retried request, a redelivered queue message, a crawler that fetches a URL it already stored. You have a key. Put a unique constraint on it.

For requests, the key is the idempotency key. For events, it is the event id. For a crawler, it is the canonical URL plus a content hash, so you can tell “same URL, new bytes” from “same URL, same bytes.” The insert that records “I have seen this” commits with the side effect. A filter in front (if key in redis) is an optimization, not the guarantee. Redis can evict. The database constraint cannot.

Keep the record at least as long as duplicates can still arrive. A crawler that revisits forever can store the hash and not the body. A payment retry window is a day, not forever, but it is not five seconds.

Near duplicates

Profiles, web pages, support tickets, and training data are duplicates without sharing a key. All-pairs comparison is quadratic and you should say the number so the interviewer hears you refuse it.

Block first. Exact normalized email or phone. Same domain and title. Same geohash and business name. Blocks are allowed to miss some true pairs; they must be cheap.

Candidate generation inside a block, or across the corpus with MinHash / simhash / an embedding neighborhood, produces pairs that might match. Scoring sorts those pairs into definite-yes, definite-no, and unsure. Spend the expensive check (a human, a model call, a slow feature) only on unsure. That is how a budget of 50,000 model calls can police a million records.

Merges

Union-find is the right structure for “these ids are the same entity.” It is also how errors spread. Keep the evidence on each edge. Do not collapse a cluster of thousands because of a chain of weak links. Prefer a survivor record and aliases over deleting the duplicates, so you can undo.

For streaming dedup (ads, clicks), a bloom filter or a short-lived cache can say “probably new” and a durable unique key behind it says “definitely.” The filter controls cost. The key controls correctness. Say which decision is allowed to be wrong.