How a model learns

Strip away the mystique and a neural network is a function with knobs. You feed it an input, it multiplies that input by a big pile of numbers — the knobs, called weights — and out comes a guess.Learning is the process of nudging those knobs until the guesses stop being wrong. That is the whole game. Everything else — CUDA kernels, trillion-parameter models, the GPU bill — is this one idea, scaled up until it needs a datacenter.

We are going to earn that sentence, not just assert it. By the end of this page you will have fit an actual model to actual data by hand, seen the landscape that fitting happens on, watched an algorithm walk down it, and felt precisely how one number — the learning rate — decides whether training converges or explodes. Andrew Ng opens CS229 Lecture 2 with exactly this example, because everything in modern ML is a variation on it.

0.1 The learning problem, made concrete

Forget neural networks for a moment. Here is the simplest possible model with knobs: a straight line. It has exactly two parameters — a slope w and an intercept b — and it predicts an output ŷ from an input x:

The scattered dots below are data — pairs of (x, y) we actually observed. The line is our model. Drag w and b and try to make the line pass through the cloud of points. The short amber sticks are residuals — the vertical gap between what the model predicts and what really happened. A good fit makes those sticks short. How short, as one number, is the loss.

The standard way to turn "how wrong" into one number is the mean squared error: square every residual (so over- and under-shooting both count, and big misses hurt more than small ones), then average:

Press let it learn above and the line drives itself toward the best fit — the loss number falling as it goes. That motion is the entire subject of this chapter. But notice the machine did something subtle: from wherever the line started, it knew which way to turn each knob to reduce the loss. Where did that direction come from? To answer that, we have to stop thinking about the line and start thinking about the two knobs as a place you can stand.

0.2 Every setting of the knobs is a point on a landscape

Here is the mental leap that makes all of optimization click. The line lives in data space — the plot you just dragged. But the loss lives in parameter space: a map whose two axes are the knobs themselves, w and b. Every point on that map is one complete setting of the model — one line — and its height (here, its color) is how much loss that line has. Fitting the model is nothing more than finding the lowest point on this map.

Below is that map for the exact same dataset. Dark teal is low loss (good), murky dark is high loss (bad). The ✦ marks the true minimum — the best line that exists. Drop a starting point anywhere and press play: gradient descent walks it downhill, step by step, to the bottom. Because squared error over a line is a perfectbowl (it is convex), every path finds the same basin — a luxury we lose the moment the model gets interesting.

Toggle stochastic (SGD) and the smooth glide turns into a drunken stagger that still, roughly, heads downhill. That is not a bug — it is the single most important trick in large-scale training, and we will come back to it in 0.5. First, the engine that computes "downhill."

0.3 The gradient: the direction of steepest ascent

Standing at a point on that landscape, which way is downhill? Calculus answers this exactly. In one dimension, the derivative dL/dθ is the slope of the curve — rise over run — and its sign tells you which way tips down. In many dimensions, you take the slope separately along each knob (apartial derivative), and stack them into a vector called the gradient:

The gradient is the compass needle of learning. It points in the direction of steepest ascent — the fastest way up. So to go down, we step the opposite way. That single fact is the update rule that trains every model ever built:

θ the knobsη the learning rate — how big a step∇L the slope — which way is up

Why subtract, and why does a small step help at all? A first-order Taylor expansion says that near your current point, moving by a small Δθ changes the loss by about ∇L · Δθ. ChoosingΔθ = −η∇L makes that change −η‖∇L‖², which is negative as long as η is small enough — so the loss is guaranteed to drop. Step too far and the linear approximation stops holding, and the guarantee dies. Hold that thought.

Below, in one dimension so you can see the slope directly: the ball sits at the current knob θ; the dashed amber line is the tangent — the gradient made visible; the green arrow is the downhill direction it produces. Press play to watch θ ← θ − η∇L fire over and over. Then switch landscapes and start breaking things.

0.4 The learning rate is the knob that bites

Of every dial in this chapter, η is the one that makes or breaks training. Too small and descent is correct but glacial — thousands of steps to crawl anywhere. Too large and each step overshoots the valley floor, landing higher up the far wall; overshoot enough and the loss climbs to infinity — it diverges. There is a narrow, precious band in between, and finding it is half the practical job of training.

For a simple quadratic bowl there is even an exact threshold: descent converges only whileη < 2/λ, where λ is the curvature (how sharply the bowl bends). The plot below runs the same descent at several learning rates at once, tracking loss versus step on a log scale. Slide yourη and watch the bold curve: it either sinks toward zero (converging), zig-zags down (oscillating), or launches off the top (diverging).

0.5 Batch, stochastic, and mini-batch

There is a hidden cost in the gradient. The clean formula for ∇L averages over everytraining example — that is batch gradient descent, the smooth path in the surface demo. With a dataset of billions of tokens, computing the exact gradient for a single step is absurdly expensive. So we cheat, productively:

• Batch GD — use all n examples per step. Exact direction, ruinously slow per step.
• Stochastic GD (SGD) — use one random example per step. Noisy, jittery, but cheap and fast — the stagger you saw when you toggled the surface demo.
• Mini-batch GD — use a small handful (32, 256, …). The winning compromise, and what everyone actually uses.

The remarkable part: the noise is often a feature. A jittery gradient can rattle the ball out of a shallow bad valley and into a better one — a free approximation of exploration. And there is a systems reason mini-batches won: a batch of examples is a matrix, and a GPU eats matrix multiplies far faster than it processes examples one at a time. The batch size is chosen as much by the hardware as by the math — a theme that will dominate the second half of this course.

0.6 When the landscape isn't a bowl

The line-fitting bowl was convex — one basin, one answer, reachable from anywhere. Real networks are wildlynon-convex: their loss landscapes have many valleys, long flat plateaus, andsaddle points (uphill one way, downhill another) where the gradient nearly vanishes and descent stalls. Go back to the 1-D demo and switch to Two minima: gradient descent isgreedy — it only ever walks downhill from where it stands — so where you start decides where you land. Drag θ₀ left and it settles in the left valley; start right and it finds the right one. Neither knows the other exists.

Now try Plateau: across the flat middle the slope is almost zero, so the steps become tiny and progress crawls — exactly the trap that momentum and adaptive optimizers (Adam) are built to escape by building up speed across flat stretches. Non-convexity is the reason initialization, momentum, and learning-rate schedules all exist. We meet those tools head-on inModule 4. For now the takeaway is humbler and deeper: there is no guarantee of the global best — only a reliable way to get reliably better. Astonishingly, at the scale of real networks, reliably-better turns out to be more than enough.

0.7 Forward and backward — where the two costs live

Every training step has two halves, and it is worth naming them now because the entire GPU story hangs on the difference:

• Forward pass — run the input through the model to produce a guess and compute the loss L. "How wrong am I?"
• Backward pass — work backwards through the model computing ∇L for every knob by the chain rule. "Which way should each knob move?"

In our two-knob toy we could write the gradient by hand. In a real network, backpropcomputes millions to billions of these partial derivatives, layer by layer — the subject ofModule 3. Both passes are, underneath, enormous piles of matrix multiplies, and matrix multiply is precisely what a GPU is built to do fast. Which is the whole reason this project descends the stack from here.