Dropbox

What you are designing

A user drops files in a folder. Other devices, and people they shared with, end up with the same bytes. Files can be large. Clients go offline. Assume tens of millions of users, files up to a few gigabytes, and that most edits touch a small fraction of a file.

Requirements

• Upload, download, rename, delete, and nest files in folders.
• Sync changes to other devices within a few seconds when they are online, and catch up after being offline.
• Share a file or folder with another user at a permission (view, edit).
• Do not lose a conflicting edit.

Bandwidth matters more than shaving a millisecond off the API.

Entities and API

User, Node (file or folder: name, parent, owner), Version (a list of chunk hashes), Chunk (hash, size, location), Change (user, version, cursor), Acl.

POST /v1/files/uploads                 -> upload id + part URLs
POST /v1/files/{id}/commit             { chunk_hashes } -> version
GET  /v1/changes?cursor=               -> changes, next cursor
POST /v1/shares                        { node_id, grantee, role }

High-level design

flowchart LR
  D1[Device] -->|chunks| Blob[(Object storage)]
  D1 -->|commit metadata| API
  API --> Meta[(Postgres metadata)]
  API --> Feed[Change log]
  D2[Other device] -->|poll or SSE cursor| API
  API --> Blob

The client splits a file into chunks (say 4 MB), hashes each, and asks which hashes the server already has. It uploads only the missing ones via presigned multipart URLs. Commit writes a new version row pointing at the ordered hashes and appends a change for every device that can see the file.

Data model

CREATE TABLE nodes (
  id          BIGSERIAL PRIMARY KEY,
  owner_id    BIGINT NOT NULL,
  parent_id   BIGINT,
  name        TEXT NOT NULL,
  is_dir      BOOLEAN NOT NULL,
  head_ver    BIGINT,
  UNIQUE (owner_id, parent_id, name)
);
CREATE TABLE versions (
  id        BIGSERIAL PRIMARY KEY,
  node_id   BIGINT NOT NULL REFERENCES nodes(id),
  chunks    TEXT[] NOT NULL,
  created_by BIGINT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE changes (
  user_id  BIGINT NOT NULL,
  seq      BIGINT NOT NULL,
  node_id  BIGINT NOT NULL,
  ver_id   BIGINT,
  PRIMARY KEY (user_id, seq)
);

Chunks themselves are objects at key = hash. A reference count or a garbage-collection scan reclaims unreferenced hashes. Do not delete on the user-facing path.

Deep dives

Sync. Each user has a monotonic seq. A device stores the last seq it applied. GET /changes returns the page after that seq. SSE can poke the device to pull. Operations are small (the version id, not the bytes). Downloads of missing chunks happen after the metadata is applied.

Conflicts. If commit includes a base version and the server’s head is different, do not overwrite. Create a conflicted copy (notes (conflicted copy from laptop).txt) or return 409 and let the client do it. Last-write-wins is acceptable only for a key-value note you have called out as such.

Sharing. An ACL row grants a user or a group access to a node. The change fan-out writes a changes row per recipient. Folder sharing includes descendants; decide whether permission is inherited at read time (simpler, slower) or materialized (faster, harder to revoke). Revocation deletes the grant and stops new presigned URLs. Already-downloaded bytes cannot be unretrieved; say that.

Scale. Metadata is relational and sharded by owner_id once one database is not enough. Blobs are content-addressed and live in object storage with a CDN for downloads. The hot problem is the change feed, which is an append-only index per user, not the bytes.

Levels

• Mid-level. Metadata versus bytes, presigned upload, a way for the other device to learn about the change.
• Senior. Chunking and dedup, cursors, conflict behavior, ACLs that actually get checked before a URL is signed.
• Staff. Garbage collection of chunks, sharing revocation limits, sharding the metadata by tenant, and offline clients that return with a week’s edits.