Systems & the GPU

A GPU is not a fast CPU — it is a throughput machine. Instead of a few clever cores that race through one instruction stream, it has thousands of simple lanes that all run the same instruction on different data, and it hides the long wait for memory by keeping many jobs in flight at once. The entire discipline of making models fast is the discipline of keeping those lanes fed. To do that you need three pictures in your head: how the lanes are organized, how far the data has to travel to reach them, and — the punchline — a single ratio that predicts which of those two is your bottleneck.

13.1 Threads, warps, blocks, SMs

You write a kernel — a small program for one thread — and launch it across a grid of thousands of threads. The hardware does not schedule threads one at a time. It bundles them into warps of 32 that execute a single instruction together: SIMT, single instruction, multiple threads. Warps are grouped into blocks, and each block is pinned to one streaming multiprocessor (SM) — the GPU's fundamental unit of execution, of which a modern chip has around a hundred. Threads in a block can share a fast scratchpad and synchronize; threads in different blocks cannot.

This organization buys latency hiding. When one warp stalls waiting on memory, the SM instantly swaps in another warp that is ready to compute, so the arithmetic units rarely sit idle — provided there are enough warps resident. The number an SM can keep in flight is its occupancy. But occupancy is a means, not the goal: an SM that is fully occupied issuing instructions can still be delivering a tiny fraction of its arithmetic, because most of those instructions are loads, not math. Keep that in mind — it is the seed of the endgame.

There is a second, quieter tax. Because a warp shares one instruction pointer, an if that sends some lanes one way and the rest another forces the warp to run both paths in sequence, masking off the lanes that are not on the current path. This is warp divergence. Split a warp into D different branches of equal length and you do D passes with only 32/D lanes doing useful work each pass — SIMT efficiency drops to 1/D. Drag the branch count below and watch the warp light up one masked pass at a time.

13.2 The memory hierarchy

Compute is cheap; moving data is expensive. Every value a thread touches lives somewhere on a steep ladder, and each rung down is roughly an order of magnitude slower — and larger — than the one above it:

  registers        ~20 TB/s     per-thread, instant           tiny
      │
  shared / L1      ~10 TB/s     per-block scratchpad          ~KB
      │
  L2 cache         ~5  TB/s     shared across SMs             ~MB
      │
  HBM (global)     ~3  TB/s     the whole model lives here    ~GB   ◄ the wall
      │
  host / PCIe / NVLink   ~0.03–0.9 TB/s   off-chip            ~TB

The number that governs a kernel's fate is HBM bandwidth — how fast weights and activations stream from global memory into the SMs. A kernel that reads a lot of memory per unit of math is limited by this pipe long before it troubles the compute units. The whole art of a fast kernel — fusing operations, tiling into shared memory, reusing every loaded value — is about climbing this ladder: touch HBM once, then do all your work out of registers and shared memory. Move the payload slider and watch how long the same bytes take to arrive from each rung.

That is the difference between compute-bound and memory-bound, and there is a clean way to predict which one you are — before you ever run the kernel.

13.3 Arithmetic intensity & the roofline

Arithmetic intensity is the single ratio that governs everything: the floating-point operations a kernel performs divided by the bytes it moves from memory.

A GPU has two hard ceilings: peak compute π (FLOP/s) and peak memory bandwidth β (byte/s). The most performance you can attain at a given intensity is whichever ceiling you hit first — the memory-limited slope, or the compute roof:

The two regimes meet at the ridge point — the knee of the roof. Left of it your kernel is memory-bound; right of it, compute-bound. Its intensity is simply the ratio of the two ceilings:

Below is the roofline, live. The bent line is the ceiling: a rising memory slope that flattens into the compute roof at the knee. Drag the workload dot left and right to change its arithmetic intensity — it rides up the memory slope, then hits the flat roof. The knobs set the chip's two ceilings. Watch % of peak: anywhere left of the knee, the lanes are starved and most of the GPU's arithmetic is simply unreachable, no matter how busy the chip claims to be.

The labelled ticks are real workloads. An elementwise add or a LayerNorm moves a lot of bytes for almost no math — intensity well under 1, hopelessly memory-bound. Attention decode — generating one token at a time from the KV cache — reads the whole cache to do a sliver of arithmetic, so it too lives on the memory slope. Only a big batched GEMM (matmul) reuses each loaded value across many multiply-adds, pushing intensity past the knee where the compute roof finally pays off. This is why serving is memory-bound and training large batches is compute-bound — same chip, opposite corner of the roofline.

13.4 Three ways to be slow

The roofline names two bottlenecks; in practice there is a third. Following Horace He's framing, every kernel's wall-clock time is set by the largest of three independent costs, and speeding up anything but the dominant one buys you nothing:

• Compute-bound — the math units are the limit. Time ≈ FLOPs / π. This is where you want to be: big matmuls, large batches, training.
• Memory-bound — the HBM pipe is the limit. Time ≈ bytes / β. Elementwise ops, normalizations, and single-token decode all live here.
• Overhead-bound — the kernel is so small that launch latency and Python dispatch dominate, and the GPU is barely doing anything at all. Time ≈ a fixed few µs, independent of the work.

Pick an operation and sweep its size. Tiny problems are overhead-bound no matter what they are — the fixed launch cost swamps the work. Grow them and each op reveals its true nature: the low-intensity ops flatten into a memory-bound regime, while the GEMM climbs into compute-bound territory. The tallest bar is the one you must attack; the other two are a distraction.