Pretraining & scaling laws

The training objective for a base LLM is astonishingly simple — the same one from the very first chapters, scaled up. Take an ocean of text, and at every position ask the model to predict the next token, scoring it by cross-entropy. No labels, no human annotation: the text is the supervision. What turns this from brute force into engineering is that the results are predictable — you can forecast the loss of a run you have not done yet.

10.1 One objective, an ocean of data

The corpus is web text, books, and code — trillions of tokens, aggressively deduplicated and filtered for quality, because repeated or junk data wastes the budget. Training marches through it in batches: a forward pass predicts the next token everywhere at once, backprop computes the gradient, and the optimizer takes a step. The one dial that most decides whether the run succeeds is the learning rate, and it is not held constant — it is scheduled: a short linear warmup from zero (so the fresh, random model isn't blown apart), then a slow cosine decay back toward zero (so it settles into a good minimum).

Below, the loss descends as more tokens are seen (note the log axis), with the learning-rate schedule drawn faintly behind it. Slide the peak learning rate: too low and the descent is sluggish; near the sweet spot it drops fastest; push it too high and the run diverges — the loss climbs instead of falls, the failure every practitioner has watched happen at 3 a.m.

10.2 Loss is a power law

The central empirical discovery: test loss falls as a power law. Make the model bigger and loss drops along a straight line on a log-log plot; add more data, same story. There is an irreducible floor — the entropy of language itself — that you approach but never beat. Hoffmann et al. captured both effects in one equation, with parameters N and training tokens D:

E is the irreducible loss; the second term is error from a model too small to represent the pattern; the third is error from too little data to pin it down. The magic is in the shape. Plot the raw loss and you get a curve bending down toward the floor E — unremarkable. But subtract the floor and plot L − E on log-log axes, and it becomes a dead-straight line whose slope is the exponent α ≈ 0.34. Toggle between the two views:

The exponents are gentle — you need multiplicative increases to keep gaining — which is why frontier training runs chase order-of-magnitude jumps in compute rather than incremental ones.

10.3 The compute-optimal knee

Compute is the real constraint, and to good approximation a training run costs C ≈ 6·N·D FLOPs. That ties the two together: for a fixed budget C, every extra parameter must be paid for with fewer tokens, and vice-versa. Spend it all on a huge model and it never sees enough data; spend it all on data and the model is too small to use it. Between those failures sits a minimum — the compute-optimal point. Slide the compute budget below and watch the loss curve and its optimal knee slide with it. Drag the model size along the curve to feel the trade — Chinchilla's headline was that GPT-3 and its peers sat far to the right of this knee: too big, too undertrained.

10.4 Emergence: when the curve lies

Cross-entropy loss falls smoothly and predictably. But some downstream abilities don't: a task like multi-digit arithmetic or following a chain of reasoning can sit near zero across many orders of magnitude of scale, then jump to high accuracy once the model crosses some size — Wei et al. called these emergent abilities. The smooth loss gave little warning. Slide scale below and watch the two curves diverge: the loss slides gently while the task metric stays flat, flat, flat — then snaps up.

A caveat worth keeping: some of that sharpness is in the ruler, not the model. A metric like "exact-match accuracy" is all-or-nothing, so smooth underlying progress can look like a sudden jump; measured with a softer metric the emergence often stretches back into a gentle slope (Schaeffer et al.). Either way, the practical lesson stands — loss is easy to predict, capabilities are not, and that is much of why scaling stays surprising.