Convolutions & vision

Everything so far treated inputs as a flat list of numbers. That falls apart for images. A modest 200×200 photo is 40,000 pixels; a dense layer from Module 3 would need a separate weight from every pixel to every neuron — billions of knobs, and it would have to learn what a vertical edge looks like separately for every position it could appear. Theconvolution fixes both problems with one idea: learn a small pattern once and slide it across the whole image.

5.1 Why a dense layer can't see

Two things go wrong when you wire an image straight into a dense layer. First, the parameter count explodes: connect 40,000 input pixels to even a thousand hidden neurons and you have 40 million weights in the first layer alone — most of them redundant. Second, and deeper: a dense layer has no notion of position. A weight that learned to recognize a whisker in the top-left corner is useless when the cat moves to the right — the network would have to relearn the same pattern for every possible location. Images have structure a flat vector throws away: nearby pixels are related, and a pattern is the same pattern wherever it lands. The convolution is built to exploit exactly that.

5.2 The kernel that slides

A kernel (or filter) is a small matrix of weights — often just 3×3. You lay it over a patch of the image, multiply each kernel weight by the pixel under it, and add the nine products into a single output number. Then you slide it one pixel over and do it again. That is the whole operation: amultiply-accumulate — the same core move as the matmul in Module 1 — repeated at every position. The grid of outputs it produces is a feature map: a new image showing how strongly the kernel's pattern matched at each spot.

The magic is weight sharing: the same nine numbers are reused at every position, so a kernel that has learned to spot a vertical edge spots it in the top-left corner and the bottom-right corner alike. One small pattern, learned once, applied everywhere — orders of magnitude fewer weights than a dense layer, and naturally translation-aware. Below, pick a kernel and hover the image: the highlighted 3×3 window is the patch under the kernel right now, and the matching cell in the feature map is the dot product it produces. Watch edge kernels light up on boundaries and blursmear everything soft.

5.3 Stride, padding, and the shape of the output

Two knobs control how the kernel sweeps. Stride is how far it jumps between positions: stride 1 visits every pixel, stride 2 skips every other one and halves the output's size — a cheap way to downsample. Padding adds a ring of zeros around the border so the kernel can center on edge pixels; without it, every convolution shrinks the image a little, and a deep stack would erode away to nothing. "Same" padding is chosen precisely to keep the output the same size as the input.

These combine into one formula every practitioner memorizes — the output side length for anN×N input, kernel K, paddingP, stride S:

Below, set the kernel size, stride, and padding and watch the window step across the padded input, filling the output grid one cell at a time. The formula updates live with your numbers so you can see exactly where the output shape comes from — and why stride 2 halves it while "same" padding holds it fixed.

5.4 Pooling — seeing that, not where

After a convolution, a pooling layer shrinks each feature map by summarizing little neighborhoods. The common choice is max pooling over 2×2 blocks with stride 2: each block collapses to its single strongest value, so the map halves in width and height. Two things come free. First, it is far cheaper downstream — a quarter of the pixels. Second, and more subtly, it buystranslation tolerance: because pooling reports the strongest response in a region, a feature that shifts by a pixel usually lands in the same pool and produces the same output. The network learns to care that an eye appeared in a region, not its exact pixel.

Below, a bright feature sits on an 8×8 map. Each 2×2 block reduces to its max (or average — toggle it), giving the 4×4 pooled map on the right. Now drag the shift slider to nudge the feature around: watch how little the pooled output changes. That stability is what makes convnets robust to small translations — the same object, moved a little, still reads the same.

5.5 Depth, receptive fields, and the hierarchy of vision

A single 3×3 kernel sees only a 3×3 patch. But stack a second 3×3 layer on top and each of itsneurons sees a 3×3 window of the first layer's outputs — which each already summarized a 3×3 patch of the input. Compose them and the top neuron now depends on a 5×5 region of the original image. This growing window is the receptive field, and it widens with every layer: for 3×3 convolutions it is1 + 2L pixels across after L layers (pooling makes it grow far faster). Depth is how a network trades a narrow, local view for a wide, global one.

And as the window widens, the features grow up with it. Zeiler & Fergus famously visualized what each layer responds to: the first layer fires on edges and color blobs; the next ontextures and simple shapes; deeper still on object parts — an eye, a wheel, a doorknob; and the top on whole objects. Nobody programmed this ladder; it emerges because composing local detectors is the most efficient way to explain images. Drag the depth below and watch one deep neuron's receptive field spread across the input as the hierarchy climbs.

5.6 The architectures that made it work

The same sliding kernel, stacked and refined, is the whole history of computer vision.LeNet-5 read handwritten digits with a handful of conv-and-pool layers back in 1998.AlexNetscaled the idea onto GPUs in 2012 and won ImageNet by a landslide, kicking off the deep-learning era. VGG showed that just stacking small 3×3 convs, many deep, worked beautifully.ResNet then broke the depth barrier: past a couple dozen layers, plain stacks got harder to train, so ResNet added skip connections that let the signal (and the gradient) bypass a block entirely — the same vanishing-signal fix we met inModule 4, which let networks go past a hundred layers. Every one is the convolution of this chapter, wired a little smarter.