Inference & decoding

Everything up to now produces, at each step, a vector of probabilities over the vocabulary — the model's guess at the next token. Decoding is how you turn that stream of distributions into actual text. It splits into two halves: which token to choose (a statistics question) and how fast you can choose it (a systems question). The first decides whether the output reads like a robot or a rambler; the second decides whether serving it costs cents or dollars.

12.1 Reshaping the distribution

The simplest rule, greedy decoding, always takes the top token. It is fluent but flat, and it loops — "the the the." Beam search keeps the few highest-probability sequences, which helps for translation but, for open-ended text, actually makes degeneration worse: the highest-probability continuation of human text is often bland and repetitive, a phenomenon Holtzman et al. called neural text degeneration. Real generation therefore samples — but shapes the odds first, with three knobs.

Temperature T divides the logits before the softmax: below 1 it sharpens toward the top token (conservative), above 1 it flattens toward uniform (wild), and at 0 it collapses to greedy. Top-k keeps only the k most likely tokens. Top-p (nucleus) keeps the smallest set whose probability sums past p — an adaptive cutoff that trims to a few tokens when the model is confident and many when it is unsure. Whatever survives is renormalized and sampled.

Below, a real next-token distribution after the context "The ___". The faint outline is the raw model probability; the solid bars are what survives your knobs. Grayed tokens have been trimmed away. Hit sample to draw one token, or ↻ resample to generate a whole sentence from a tiny grammar under the same settings — and watch it swing from robotic to unhinged.

12.2 One token at a time — prefill vs decode

Generation is autoregressive: sample a token, append it, feed the whole thing back in, sample again. That splits a request into two phases with utterly different performance.Prefill processes the whole prompt at once — one big parallel matmul that keeps the GPU's arithmetic units busy.Decode then emits one token per step, and each step must load every weight in the model from memory just to do a single token's worth of math.

That ratio — math done per byte loaded, the arithmetic intensity — is the whole story. Prefill amortizes one weight load across hundreds of tokens; decode spends the same load on one. The KV cache keeps decode from re-reading the whole context each step, but it cannot fix the fundamental imbalance. Press play and watch the right panel: the weights load in full every single step, while the useful math collapses the moment decode begins.

12.3 Speculative decoding

If decode is memory-bound, the GPU has spare compute sitting idle — so spend it verifying guesses.Speculative decoding runs a small, fast draft model to propose γ tokens cheaply, then the big target model checks all γ in a single forward pass — the same cost as generating one token, because that pass was memory-bound anyway. It accepts the longest prefix the draft got right, corrects the first mistake, and moves on.

The beautiful part: a rejection-sampling correction makes the output identical in distribution to what the target would have produced alone — it is a pure speedup, not an approximation. When the draft agrees often (acceptance rate α high), you get several tokens per expensive pass. Raise α below and watch the tokens-per-pass — and the speedup — climb.

12.4 Serving at scale — batching

One request cannot keep the GPU fed; the fix is to make the loaded weights serve many requests at once. Stack B requests into a batch and each weight load produces B tokens instead of one — throughput climbs almost linearly until the chip finally runs out of compute (the roofline knee), where it plateaus. vLLM pushes this with continuous batching — slotting new requests in the instant an old one finishes rather than waiting for the whole batch — and PagedAttention, which stores the KV cache in fixed pages so memory doesn't fragment and more requests fit.

Drag the batch size and read the two curves against each other: throughput soars while per-request latency barely moves, until the knee. Below the knee you are wasting the GPU; above it you are wasting nothing but paying a little latency. That trade — where to sit on this curve — is the core decision in LLM serving.