The LLM assembled

We now have all the parts: token vectors,attention, the MLP, andresiduals and normalization. A GPT is just these assembled into one decoder block and stacked. Understand the block and you understand the model — the only things that change with scale are how many times you repeat it and how wide each one is. Karpathy's nanoGPT fits the whole architecture in a few hundred readable lines for exactly this reason.

9.1 The residual stream

The organizing idea is the residual stream: a vector per token that flows straight up through the network, where every sub-layer only ever adds to it. Attention reads the whole stream, decides how tokens should share information, and adds its result back. Then the MLP reads each token independently, transforms it, and adds its result back. Nothing is overwritten. Anthropic's circuits work frames this as a shared communication channel: the stream is the sum of the embedding plus every block's contribution, and blocks "talk" to later blocks by writing directions into it.

Because everything is an addition onto a common stream, gradients have a clean path all the way down — the same residual trick that let ResNet go deep. Below, five feature channels of the stream are plotted as they rise through eight blocks; each block nudges them. Click a block to ablate it — zero out its contribution — and watch the output shift. The final vector is literally the embedding plus the surviving deltas; remove a writer and everything downstream re-routes.

        x   ── residual stream (one vector per token, width d)
        │
    ┌───┴────┐
    │ LayerNorm │
    └───┬────┘
        │
  ┌─────┴──────┐
  │ Multi-head │   tokens look at each other
  │ Attention  │   (with RoPE on Q, K)
  └─────┬──────┘
        │
    x ──⊕            residual add
        │
    ┌───┴────┐
    │ LayerNorm │
    └───┬────┘
        │
  ┌─────┴──────┐
  │    MLP     │   up-project → GELU → down-project
  └─────┬──────┘
        │
    x ──⊕            residual add
        │
        x'  ── to the next block  (×N)

Note the norm sits inside each branch, before the sub-layer — this is the pre-normarrangement (Xiong et al.). It keeps the residual stream itself un-normalized and additive, which is what makes very deep stacks train stably; the original post-norm GPT needed careful learning-rate warmup to avoid diverging.

9.2 Inside the block: mix, then think

The two sub-layers do complementary jobs. Attention is the only place tokens exchange information — it mixes across positions. The MLP never looks sideways; it processes each token's vector on its own, the same learned function applied everywhere — it thinks per position. Roughly: attention decides what to gather, the MLP decides what to make of it.

The MLP is deceptively simple and holds most of the model's parameters. It up-projects the width d to 4d, applies a GELUnon-linearity, then down-projects back to d. That 4× hidden layer is where the network stores much of what it knows. GELU is a smooth gate — near-zero for negative inputs, near-identity for positive ones — so it softly switches cells on and off. Slide the bias below to push the pre-activations up or down and watch how many cells "fire."

9.3 RoPE: position as rotation

Attention has no built-in sense of order — swap two tokens and the math is identical. Early Transformers added a positional vector to each token. Rotary Position Embedding does something cleaner: itrotates each query and key by an angle proportional to its position. A token at position m has its vector turned by m·θ. The beautiful consequence is that when you take the dot product of a query at position m with a key at position n, the two rotations combine into one that depends only on the gap m − n:

That is exactly what you want: a token should care that another is three words back, not that it sits at absolute index 4,097 — and RoPE also extrapolates gracefully to sequences longer than it trained on. Below, a query and a key start at fixed base angles. Slide each token's position to rotate its vector, and watch the attention score. Shift both positions together and the score does not move — only the gap between them matters.

9.4 Stack it into a GPT — and the KV cache

Repeat the block N times (GPT-2 small: 12; the large open models: 80+), add the embedding at the bottom and a final norm plus an output projection at the top, and that is the model. The output projection is often the same matrix as the embedding, transposed — weight tying — turning "which row is this vector nearest?" back into token probabilities. Training and a single forward pass are clean.

Generation is where a subtlety bites. The model emits one token at a time, and each new token attends to all previous ones. Recomputing every earlier token's keys and values each step would be quadratic and wasteful, so they are stored — the KV cache. It grows by one column per step and is read in full on every step. Press play and watch it balloon.