Linear algebra — matmul as the unit of work

In Module 0 a model was a function with knobs, and every training step was one line: θ ← θ − η∇L(θ). That was deliberately one-dimensional — a single weight rolling downhill. Real models have billions of knobs, and they are not a loose bag of numbers. They are arranged into matrices, the data flows through as vectors, and the "run the input through the model" step is — almost entirely — matrix multiplication. This chapter builds that operation from the ground up: an arrow, a dot product, a matrix as a function that warps space, and finally the grid of dot products that eats every FLOP in a datacenter.

1.1 A vector — one object, four costumes

Strang opens 18.06 with the humblest object in the subject: a column of numbers. Everything is built from it. A vector v = [3, 2] can be read four ways, and fluency means switching between them without thinking:

• an arrow from the origin — the geometric picture (3 right, 2 up);
• a point in space — the tip of that arrow, a location;
• one data example — a house as [bedrooms, bathrooms], a token as a 768-number embedding;
• one layer's activations — the numbers flowing between two layers of a net.

Two operations define a vector space and nothing else is allowed at this level: you can addtwo vectors (tip to tail), and you can scale one by a number (stretch or flip the arrow). Every richer idea — spans, transformations, whole neural networks — is assembled out of just those two moves. Drag the arrowhead below and watch the components, the length ‖v‖ = √(x²+y²), and the scaled copy 2v update together.

Notice length is just a dot product with itself under a square root: ‖v‖ = √(v·v). That is not a coincidence — the dot product is where every notion of length, distance, and angle comes from, which is exactly the next section.

1.2 The dot product — the atom, algebraically and geometrically

Take two vectors of the same length — say an input x and one row of weights w. Thedot product pairs them up, multiplies each pair, and adds the results into a single number:

w_i the weights — one knob per inputx_i the input featuresΣ a multiply-accumulate down the length

The magic is that two completely different-looking formulas give the same number. Algebraically it's a multiply-accumulate. Geometrically it's ‖w‖‖x‖cos θ — the two lengths times the cosine of the angle between them. So the dot product is really a machine for measuring alignment:

• Positive — the vectors point the same general way (angle < 90°). The neuron "fires."
• Zero — they are perpendicular (angle = 90°). No signal; they are orthogonal.
• Negative — they point apart (angle > 90°). The neuron is inhibited.

Every "score," every similarity, every neuron pre-activation, every attention weight in a Transformer (Module 7) is a dot product asking how aligned are these two vectors? Drag either arrow below. Watch the algebraic sum and the geometric ‖a‖‖b‖cosθ stay locked to the same value, and watch the projection — the shadow a casts on b — which is what the dot product geometrically measures.

When you make the arrows perpendicular the dot product snaps to ~0 — that's orthogonality, and it's why "independent directions" in a model are ones that don't leak into each other. Two identical unit arrows give exactly 1; two opposite ones give −1. This is the same cosine similarity that ranks search results and clusters word embeddings (Module 8).

1.3 A matrix is a function that warps space

Here is the idea 3Blue1Brown builds the whole series around, and the one that turns linear algebra from bookkeeping into intuition: a matrix is a linear transformation — a function that takes in a vector and spits out a vector, by warping all of space at once. "Linear" means two promises the warp must keep: the origin stays put, and grid lines stay straight, parallel, and evenly spaced. No curving, no bending — only rotate, stretch, squish, shear, reflect.

And a 2×2 matrix is nothing more than a record of where the two basis vectors land. The first column says where î = [1,0] goes; the second says where ĵ = [0,1] goes. Because the transform is linear, that's all you need — every other vector just rides along:

[a c] where î lands (column 1)[b d] where ĵ lands (column 2)det = ad − bc = how areas scale

Play the transformation directly. The four sliders are the four matrix entries; the faint grid is the original space and the bright grid is where it lands. The teal and amber arrows are î andĵ — they are literally the columns of the matrix. The shaded parallelogram they span has area equal to the determinant ad − bc: how much the transform stretches area. When it hits 0, space has been flattened onto a line — the matrix is singular, information was destroyed, and there's no way back. Try the presets, then flatten it and watch det → 0.

This is the mental model to carry forever: a neural network layer y = Wx + b is a matrix warping the activation space (then a bias shift, then a nonlinear bend we'll add in Module 3). "Learning" isslowly adjusting how each layer warps space so that, by the last layer, cats and dogs land on opposite sides of a line. Depth = many warps composed. Which is the next idea.

1.4 Matrix multiply = composing warps = a grid of dot products

If a matrix is a function, then applying two in a row — first B, then A — isfunction composition, and the single matrix that does both at once is the productC = A·B. This is why matrix multiplication has the "strange" rule it does, and why order matters (AB ≠ BA): doing a rotation then a shear is not the same as a shear then a rotation. Strang calls this the most important operation in the subject; 3Blue1Brown calls it "the" reason the row-times-column rule exists at all.

A layer doesn't compute one dot product — it computes a whole grid at once. Stack the inputs into a matrixA (each row an example, m of them, width k) and the weights intoB (k tall, n wide). The result C = A·B has one entry forevery (row of A, column of B) pairing — and each entry is a dot product:

Below is that formula made physical. Hover any cell of C to light up the row of A and the column of B that produce it. Click it to watch the multiply-accumulate build up, one term at a time — the amber pointer sweeping A's row while the teal pointer sweeps B's column, exactly in step. Then drag the dimensions and watch the work explode.

Two things to feel in your hands here. First, every cell of C is independent — cell (0,0) needs nothing from cell (0,1). Nothing stops you computing all of them at the same time. Second, drag k, m, or n up and watch total MACs climb as their product. That's the whole story of scale in one readout.

1.5 Why it's this operation that costs everything

The MAC count for an m×k by k×n multiply is m·n·k — one per (output cell) × (length of the dot product). Since each MAC is a multiply and an add, that's about 2·m·n·k floating-point operations. The number grows with the product of the dimensions, so doubling the model width roughly quadruples the matmul cost. A single transformer layer at scale is billions of these FLOPs; a forward + backward pass stacks hundreds of such matmuls; training repeats that trillions of times. ~90% of all the arithmetic in modern deep learning is matrix multiply. Optimize this one kernel and you've optimized the field.

1.6 Why the GPU exists at all

Go back to viz 4 and press compute all. Watch C fill in as a wavefront. On a CPU those cells would be computed a few at a time, in sequence. But you already noticed they're independent — so what if you had thousands of tiny arithmetic units and handed one output cell to each? That is precisely a GPU: not a fast brain but a very wide one — thousands of MAC lanes running the same dot-product recipe on different data at once. Matrix multiply is embarrassingly parallel, and the GPU is the machine shaped exactly to that fact.

• A CPU has a few very fast cores — great for branchy, sequential logic.
• A GPU has thousands of simple lanes grouped into Streaming Multiprocessors (SMs) — great for doing the same math on mountains of data.
• A matmul maps onto that grid almost perfectly: tile the output, give each tile to an SM, each lane a cell. This is what the die below is doing — the whole of Module 13 is how that mapping succeeds or stalls.