Alignment

A base model from pretraining is a magnificent autocomplete: it continues text the way the internet would. Ask it a question and it might answer — or it might list ten more questions, or drift into an advertisement, because all of those are plausible continuations. Alignment turns that raw predictor into something that actually tries to be helpful, honest, and harmless. The trick is that we can rarely write down what "good" means as a loss function — but people can reliably compare two answers and say which is better. Alignment is the machinery that turns those comparisons into a trained model.

11.1 The alignment pipeline

The classic recipe — InstructGPT — runs in three stages.Stage 1, SFT: collect high-quality example dialogues and fine-tune the base model on them with the same next-token loss as pretraining. This teaches the format: answer the question, follow the instruction, adopt a helpful tone. Stage 2, reward modeling: sample pairs of responses, ask humans which they prefer, and fit a reward model that scores any response the way people would. Stage 3, RL: optimize the SFT model — now the policy — to earn reward, using PPO, while a leash keeps it close to where it started.

The reason for the middle step is subtle and important: judging "which of these two answers is better" is far easier and more consistent for a human than writing the ideal answer from scratch. Comparisons are cheap, reliable, and scalable; demonstrations are expensive and cover only what annotators happen to think of.Constitutional AI pushes this further — it replaces many of the human labels with an AI judge told to follow a written constitution (RLAIF), so preference data can be generated at machine speed.

Watch the pipeline below. Each stage lists the model copies it must hold in GPU memory at once — the detail that makes alignment so expensive. Flip between RLHF and DPO: PPO juggles up to four models, while DPO (11.4) needs only two.

11.2 Learning a reward from comparisons

How do you turn "annotator preferred A over B" into a number? The standard answer is the Bradley–Terry model. Give every response a latent scalar reward r, and assume the probability a human prefers A over B is a sigmoid of the reward gap:

The reward model is just the transformer with a scalar output head, and it is trained by plain logistic regression on the comparison pairs — maximize the log-likelihood of the observed choices, which is the loss

Two consequences fall straight out of the sigmoid, and the widget below makes both visible. Only the gap matters — adding a constant to every reward changes nothing, so reward is defined only up to an offset. And with enough votes, the gap that best explains them is exactly Δr* = log(n_A / n_B): the reward model is doing logistic regression on the tally. Cast votes, then press fit to slide the reward to the value that best explains them.

11.3 Optimize with a leash

Now push the policy up the reward — but with a restraint. Chase reward too hard and the model finds degenerate answers that fool the reward model while being gibberish; the reward is only a proxy, and proxies break when over-optimized (reward hacking, a.k.a. Goodhart's law). So the RLHF objective adds a KL penalty that keeps the tuned policy close to the reference (post-SFT) model:

In practice this is optimized with PPO, a policy-gradient method — but the objective itself has a beautiful closed-form optimum. The best possible policy simply reweights the reference by the exponentiated reward:

Below is exactly that formula, live. The faint curve is the reference policy over a space of possible responses. Drag the reward handles to say which responses are good, and the solid curve — the aligned policy π^* — shifts its mass toward them. Turn the β leash down and it chases reward aggressively; turn it up and it barely dares to move.

11.4 DPO: skip the reinforcement learning

Classic RLHF trains the reward model, then runs PPO to push the policy up the reward — a finicky loop with several models in memory at once. Direct Preference Optimization makes a sharp observation: since the optimal policy is the closed form above, you can invert it to write the reward asr(y) = β log( π(y) / π_ref(y) ) plus a constant, drop that into the Bradley–Terry loss from 11.2, and the reward model vanishes. What remains is a plain classification loss on preference pairs — no reward model, no sampling, no RL:

In words: raise the policy's probability on the preferred answer y_w and lower it on the rejected one y_l, each measured relative to the frozen reference — with β playing the same leash role. Run the gradient below: the winner bar climbs above the reference line, the loser bar sinks below it, the margin β·(log-ratio_w − log-ratio_l) widens, and the loss falls. Push too hard and the loser's probability collapses toward zero — DPO's own over-optimization failure mode.