Probability — where the loss comes from

Module 0 rolled a ball down a loss landscape. Module 1 showed the matmul that computes a model's raw scores. But we skipped the obvious question: where does the loss come from, and why is it shaped the way it is? The answer is probability. A model's real job is not to emit one number — it is to place a bet, to output a whole distribution over what it thinks is likely. Training then asks the same question a trillion times: given the data we genuinely saw, which knob settings make that data most probable? Answer that and the loss function writes itself — and, as CS229 shows in its very first probabilistic derivation, mean-squared error falls straight out.

2.1 A model outputs a distribution, not a number

Picture a single measurement — a pixel's brightness, a sensor reading, tomorrow's price. Instead of one guess, describe your belief as a curve over all the values it could take. The workhorse curve is the Gaussian (the "normal" distribution, the bell), and it has exactly two knobs: the center μ and the spread σ. Its height at a value x is the density — how much belief sits near x:

μ the center — where the bell sitsσ the spread — how wide the belief isarea = 1 total belief always sums to one

Two things matter more than the formula. First, the area under the whole curve is always 1— you have exactly one unit of belief and the curve only decides how to spread it. Second, density is not the probability of an exact point; it is how concentrated belief is there. A tall, narrow bell (small σ) says "I'm confident it's near μ"; a low, wide bell says "could be anywhere." Confidence is a knob. That's why σ also carries the famous 68–95–99.7 rule: about 68% of belief lands within ±1σ of the center, 95% within ±2σ, 99.7% within ±3σ — the shading you'll see under the curve below.

2.2 Likelihood — scoring the knobs against reality

Now flip it around. You are handed real data points and asked: how good are these knobs? For one point, the curve's height there is its score — high if the point lands under the fat middle of the bell, low if it's stranded out in a thin tail. For the whole dataset, multiply every point's height together. That product is the likelihood — the probability of having drawn exactly this data under these knobs. Because multiplying many small numbers underflows, we take the log, which turns the product into a friendly sum:

Below is that idea you can grab. The bell is the model; the dots on the axis are real data. Each point grows a stick up to the curve — tall and green when the knobs explain it well, short and amber when they don't. Drag μ and σ to make the total as tall as possible. Then press snap to best fit to jump to the knobs that maximize it — and click the plot to drop in a new data point and watch the fit react.

Notice what your hands just learned. Slide μ off to the side and every stick shrinks — the loss shoots up. Shrink σ too tight and the far points fall off the thin tails and get punished; widen it too much and even the central points sit low under a flat curve. There is exactly one best setting, and snap to best fit lands on it. That best setting has a name: the maximum-likelihood estimate, and for a Gaussian it's just the data's mean and standard deviation.

2.3 Maximum likelihood is your loss — and MSE is a Gaussian

Maximizing a likelihood and minimizing a loss are the same act seen from two sides. Flip the sign of the log-likelihood and you get the quantity training actually drives downhill — the negative log-likelihood (NLL), the loss from Module 0:

Here is the punchline that makes probability the root of the whole field — the derivation CS229 does in its opening lectures. Take the Gaussian, hold σ fixed, and expand−log p(x). The constants log(σ√2π) don't depend on μ, so they fall away, and what's left is (x − μ)² / 2σ². Summed over the data:

That sum Σ(xᵢ − μ)² is mean-squared error. MSE was never an arbitrary choice — it is exactly what you get when you assume your errors are Gaussian and ask for the most likely knobs. And because it's a sum of squares in μ, the loss is a perfectparabola — a bowl, the same convex bowl the ball rolled down in Module 0. Dragμ below and watch the ball ride the NLL bowl; its lowest point sits exactly over the data's mean. Widen σ and the bowl flattens (the model is less sure, so being off costs less); tighten it and the walls steepen.

2.4 Surprise — why the log, and why −log p

Where does the −log come from, really? Information theory gives it meaning: −log pis surprise (Shannon called it "self-information"). An outcome the model was sure of (p → 1) carries zero surprise — you learned nothing, you already knew it. An outcome it thought impossible (p → 0) is infinitely surprising — a slap in the face. Training to minimize −log p(data) is training the model to be as un-surprised by reality as possible. Drag the probability the model assigned to the thing that actually happened, and watch the surprise the loss charges it:

2.5 The discrete cousin: softmax & cross-entropy

Bell curves are for continuous values. For a choice — which digit, which class, which next token — the model instead outputs one probability per option. A layer produces raw scores called logits(a matmul, from Module 1), and softmax exponentiates them (making everything positive) and normalizes (making them sum to 1): a proper distribution over the options.

The loss looks only at the option that was actually correct and reports−log(probability the model gave it) — the surprise curve you just dragged, applied to the true class. Confident and right → near-zero loss; confident and wrong → a large one. That iscross-entropy, and it is, once again, a negative log-likelihood. Twist the logits below, pick which class is truly correct, and watch softmax redistribute the belief while cross-entropy scores it. Every single next-token an LLM predicts during training is graded by exactly this rule, billions of times.

Play the two failure modes. Pile all the logit mass on the wrong class while the truth sits low — the loss explodes toward the steep left wall of the surprise curve. Now spread the logits flat (all equal): softmax returns 1/4 for everyone and the loss is −log(0.25) ≈ 1.39 — the "I have no idea" baseline. Good training drives the true class's bar toward 1 and the loss toward 0.