The neuron & backprop

We now have every piece: a model is knobs (Module 0), the arithmetic is matmul (Module 1), and the loss is a negative log-likelihood (Module 2). This chapter wires those pieces into the object that started the revolution — the neural network — and shows the single algorithm that trains it: backpropagation. It is nothing more than the chain rule from calculus, applied with careful bookkeeping, at scale. Understand one neuron and one backward pass, and the rest is repetition.

3.1 One neuron: sum, bias, bend

A single neuron takes a vector of inputs, multiplies each by a weight, adds them with a bias, and passes the result through a non-linear activation. The weighted sum alone is just the dot product from Module 1 — a straight line. The activation is the crucial twist: without it, stacking layers collapses back into one linear map, and the network can only ever draw a flat hyperplane.

w the weights — one knob per inputb the bias — shift the whole sumφ the activation — the bend (tanh, ReLU, sigmoid)

Three bends dominate. Sigmoid squashes to (0, 1) — a soft on/off switch, handy as a probability. tanh is its zero-centered cousin, squashing to (−1, 1).ReLU — max(0, z) — simply clips negatives to zero; it is nearly free to compute, doesn't saturate for positive inputs, and is the modern default in hidden layers. Grab the weight and bias below and watch a straight pre-activation line get bent by each. The weight sets steepness; the bias slides the bend left or right:

3.2 Why depth buys shape: universal approximation

Stack many neurons that all read the same inputs and you have a layer — every neuron's weighted sum computed at once is one matrix multiply. Feed a layer's outputs into another and you have a network: input → hidden → output. The hidden layer is where the power lives, and there's a theorem behind it: the universal approximation theorem says a single hidden layer with enough bent neurons can approximate any continuous function to any accuracy. The intuition is visual — each hidden neuron contributes one bend, and the output layer adds the bends up. Enough bends, placed and scaled right, trace any wiggle.

Watch it happen. The faint curve is a target function; the bold curve is a 1 → H → 1 tanh network trying to match it. Press fit and gradient descent tunes the bends; raiseH and the network gains the capacity to trace finer detail. Too few neurons and it can only manage a crude sketch — not enough bends to go around.

3.3 The forward pass, then backprop — follow the gradient

Forward: push inputs through the layers to a prediction, then score it against the truth with a loss. Backward: we need to know how nudging each weight would change the loss — the gradient ∂L/∂w. Backprop computes all of them in one sweep by applying thechain rule from the loss back to the inputs, and its trick is reuse: each node multiplies the gradient arriving from its right (the "upstream" gradient) by its own localderivative, then passes the product left. Nothing is recomputed.

Below is a real computation graph for one neuron scoring one example:p = w·x, z = p + b, a = tanh(z), and lossL = (a − t)². Each node shows its forward value on top. Pressbackward pass and watch the gradient flow right-to-left — the caption shows the exact local-derivative multiply at each hop. Then drag w orb, or press gradient step, and watch L fall as the weights move against their gradients.

That is the entire algorithm. A real network is this same graph with millions of nodes; the backward pass visits each once, multiplying upstream gradient by local derivative. Take one step downhill —w ← w − η·∂L/∂w with learning rate η, exactly the ball-roll of Module 0 — and repeat.

3.4 The whole loop, live: bending a boundary

Now the full thing on 2-D data. The plot is the network's mind: each pixel is colored by the class it predicts, teal vs amber, with a pale band at the undecided boundary. The dots are real data — a ring of one class around a core of the other. Press trainand every frame runs forward-and-backward over all points and steps the weights. RaiseH for more bends; push η too high and training thrashes. Then flip hidden layer → off and watch a single straight line fail to wrap a circle.

With the hidden layer gone the model is a single straight line, and a straight line simply cannot wrap a circle — the accuracy stalls near 50% no matter how long you train. That failure is the whole reason depth exists, and the reason every network from here on is many layers deep.