← Study Lab
Module 14updated 2026-07-04

Observability — the endgame

This is where all fourteen chapters land. A GPU can report 100% utilization while delivering a small fraction of its arithmetic, because the number everyone watches — nvidia-smi utilization — only asks whether a kernel was running, not whether it did useful work. The honest metric is Model FLOPs Utilization, and the two can disagree wildly. This chapter makes the gap live: pin the utilization meter at 100 and watch real throughput collapse underneath it. Closing that gap, joined from kernel to cluster, is the company.

Every chapter so far ended at the same place — arithmetic on silicon — and the last one gave us the honest axis: the roofline. This chapter turns that axis into a measurement, and in doing so states the thesis of the whole company in one sentence: the number the entire industry uses to decide whether a GPU is working does not measure whether the GPU is working.

The whole chapter in four lines1. The lie — nvidia-smi utilization is the fraction of time a kernel was resident, not the fraction of peak FLOPs delivered.
2. The truth — Model FLOPs Utilization (MFU) divides useful arithmetic by the chip's ceiling; a memory-bound run pins util at 100% and MFU in the single digits.
3. The trace — knowing MFU is 8% is worthless unless you can point at which kernel, op, layer, and node the waste lives in.
4. The bill — that gap is GPU-hours you rent and never use; at fleet scale it is millions of dollars a year, invisible on every dashboard.

14.1 What the meter actually counts

Open nvidia-smi, read utilization.gpu, see 100%, feel good. But NVML defines that field as the percent of the last sampling window during which one or more kernels was executing. A kernel that spends its whole life stalled on HBM — issuing memory loads and waiting — is still "executing." So the meter reads full while the compute units idle. It is apresence detector, not a work detector:

utilnvidia-smi=time a kernel was residentwall-clock time≈100%

The strip below is one sampling window, cycle by cycle. A green cycle issued real math; a red cycle was stalled waiting on memory — but a kernel was resident either way. The meter counts the whole strip and reports ~100%. Only the green cells did work. Slide the stall fraction up and watch the two numbers tear apart while the meter never flinches.

a memory-bound decode kernel spends most cycles here — yet stays "resident"
nvidia-smi util—cycles doing math—the illusion—

DCGM, NVIDIA's heavier profiler, does better — it exposes SM-active, tensor-core-active, and DRAM-active counters that begin to separate "busy" from "computing." But those signals stay trapped at the per-GPU level, unjoined to the op that issued the kernel or the model step it belongs to. Nobody ships the correlation.

14.2 The honest number: Model FLOPs Utilization

The metric that cannot be faked is MFU — the useful floating-point work your model actually performed, divided by what the hardware could have done at peak in the same wall-clock time:

MFU=model FLOPs / stepπpeak · step time

It is exactly the roofline read as a percentage: MFU = attained ÷ peak. A well-fed training GEMM reaches 40–60%; token-by-token decode serving often lives at 5–15%, because it is memory-bound by construction. The gap between what the util meter reports and what MFU reveals is wasted GPU-hours you are paying for and cannot see. Below it runs live — push the batch size up and watch the amber line climb off the floor as weight reuse pulls decode toward the compute roof.

nvidia-smi util—looks perfect
real MFU—actual useful work
the gap—wasted GPU-hours
how much of the time is spent on decode / elementwise vs big matmuls
bigger batches reuse loaded weights → lift decode off the memory wall
the green meter barely moves; watch the amber one

14.3 The vertical trace

Knowing MFU is 8% is a start; the money question is which 8%. That needs a signal joined vertically — from the kernel that stalled, up through the op that launched it, the model layer that owns the op, and the cluster scheduling the whole run. The interactive stack below is that trace. Every layer carries the same MFU, because the waste is the same waste seen at four zoom levels; the culprit sits at the bottom. Move the batch slider and watch the health propagate all the way up to the fleet.

the one lever that moves every layer at once

Every layer here exists in isolation today: DCGM sees the kernel counters, the framework sees the op, the scheduler sees the cluster. What no tool does is stitch them into one trace so a low fleet MFU points straight at the flash-decode kernel starving on the KV cache — and tells you the fix is a bigger batch or a different attention kernel, not more GPUs.

14.4 The bill

Here is why anyone pays for this. A rented GPU costs the same whether it runs at 55% MFU or 8% — you buy wall-clock, not work. So the gap between your MFU and a well-tuned baseline (PaLM reported ~46%) is money converted directly into heat. At a handful of GPUs it is a rounding error; at fleet scale it is a second data-center's worth of spend that shows up nowhere, because every dashboard says 100%. Set your fleet and your real MFU and read the leak.

fleet spend / day—what you rent
wasted / day—paid, never computed
wasted / year—the invisible bill
▸ this is the endgame

That stitched, vertical, honest trace — priced in dollars — is Plasmient. The wedge is the sentence at the top of this page: your GPUs are lying to you, and every chapter you just played was the proof that we can tell when they are. To measure every bit that moves you have to understand every bit that moves, from the first neuron to this kernel. Now we do.

Read the full thesis → · ← back to the map

Go deeper — the original sources

We teach it in our own words and build the demo ourselves. To go to the source, these are the lectures and texts this chapter draws on — linked, not rehosted.