LLM inference serving

What you are designing

Many clients call a completion or chat API backed by a large model you host. A request has a prompt and a max output length. You owe a latency SLO for short calls (for example first token in a few hundred milliseconds for interactive, or minutes for batch) and a cost per token you can defend. Assume GPU memory is the scarce resource, traffic is bursty, and some callers will send enormous prompts if you let them.

If you have worked on vLLM, say what that taught you about scheduling, then design the system around it rather than narrating one engine’s flags.

Requirements

• Serve POST /v1/chat/completions with streaming tokens.
• Isolate tenants so one key cannot occupy every GPU.
• Bound latency for the interactive class and throughput for the batch class.
• Fail clearly when you are over capacity.

Exact bitwise reproducibility is not the goal. Not dropping or double-billing a streamed response matters more.

Entities and API

Request(id, tenant, prompt_tokens, max_output, priority), Replica, Usage(tenant, tokens).

POST /v1/chat/completions     stream tokens, or 429
GET  /v1/usage

Streaming is SSE or chunked HTTP. The client must tolerate a disconnect by retrying with an idempotency key; you either resume (hard) or start over (say which). Starting over is the honest v1, and you bill once by key if the first stream actually finished.

High-level design

flowchart LR
  Client --> GW[Gateway: auth, token estimate, limit]
  GW --> Q[Per-tier queue]
  Q --> Sched[Batch scheduler]
  Sched --> GPU[Model replicas]
  GPU --> Client
  GPU --> Usage[(Usage ledger)]

The gateway rejects prompts over a token cap and tenants over their budget before a GPU sees them. A scheduler groups waiting requests onto replicas, preferring to fill a batch without exceeding KV memory. Replicas are stateless aside from the cache for active requests. A router can send easy calls to a small model.

Data model

You do not put prompts in Postgres on the hot path. You do put a usage ledger:

CREATE TABLE usage (
  tenant_id   BIGINT NOT NULL,
  request_id  TEXT NOT NULL,
  model       TEXT NOT NULL,
  prompt_tokens INT NOT NULL,
  output_tokens INT NOT NULL,
  PRIMARY KEY (tenant_id, request_id)
);

The unique key is how a retry does not double-bill. Queue state lives in memory plus a durable overflow if a request must survive a process crash; interactive calls usually just fail and let the client retry.

Deep dives

Scheduling. Prefill (reading the prompt) is compute-heavy. Decode (one new token) is memory-bandwidth-heavy and wants a batch of sequences. Continuous batching adds a sequence when another finishes. Cap concurrent sequences by KV bytes, which grow with context length. A 100k-token prompt is not the same job as a 200-token prompt; the admission estimate has to say so, with a buffer because the estimate can be wrong.

Tenancy and priority. Two queues: interactive with a tight SLO, and batch (evals, backfills) that soaks leftover GPU. Batch is preempted or simply not scheduled when interactive queue age rises. Per-tenant token buckets stop one customer from buying the whole cluster by accident.

Failure. A replica dies, its in-flight streams error, clients retry. Do not pin a user to a replica except as an optimization (prefix cache). Health checks must fail when the GPU is wedged, not only when the HTTP port is open. Observe queue age, time-to-first-token, tokens per second, and KV occupancy. Those four tell you if you are full.

Cost lever. Cache identical prompts (system prompt prefix). Route with a small model first when the product allows a quality drop. Batch API for offline eval with a looser SLO and higher utilization. Say the number: tokens times price, versus the user’s latency wish.

Levels

• Mid-level. Gateway, a queue, GPU replicas, a timeout, a rate limit, streaming.
• Senior. KV memory as the capacity unit, continuous batching at a high level, tenant budgets, 429 instead of an infinite queue.
• Staff. Interactive versus batch, prefix cache, what you measure, and how a model upgrade rolls out (both versions serving, then a cut).