Large blob uploads

Large files break the assumption that a request body fits in memory and finishes before a timeout. The pattern is the same for photos, video, dataset export, and resume PDFs. Only the processing after the upload changes.

Steps

1. The client calls POST /uploads with size, type, and checksum. The API checks quota and policy, creates a metadata row in state pending, and opens a multipart upload.
2. The API returns presigned URLs for parts, or a way to ask for the next part URL. Part size is a few megabytes and up, so a mobile client can retry a part without restarting a movie.
3. The client uploads parts in parallel, directly to the bucket, and keeps the etag of each.
4. The client calls POST /uploads/{id}/complete with the part list. The API finalizes the object, stores the etags and checksum, and sets state to uploaded.
5. A worker scans, transcodes, or extracts text, then sets state to ready or rejected.

Downloads of a file that is not ready return 409 or a placeholder, not the raw upload. That gap is where you hide malware scanning and where you avoid serving a half-processed video.

Failure and garbage

If the client disappears, the pending row and the incomplete multipart upload should expire (a lifecycle rule plus a status sweeper). Do not leave orphans as a billing surprise.

Completing twice must be safe. The second complete sees uploaded or ready and returns the same metadata. Part retries overwrite the same part number.

If you deduplicate by hash, scope the hash to the tenant unless you are building a public content-addressed store on purpose. A global hash lets one customer probe whether another customer uploaded a given file.

Small files

Do not force this ceremony on a 4 KB avatar. A single presigned PUT is enough. Say the threshold where you switch to multipart, rather than drawing one path for every byte.

sequenceDiagram
  participant C as Client
  participant A as API
  participant B as Object storage
  participant W as Worker
  C->>A: create upload
  A-->>C: part URLs
  C->>B: PUT parts
  C->>A: complete
  A->>W: enqueue process
  W->>A: mark ready