The mystery we're going to explain

Here's matrix-multiply throughput as a function of matrix dimension. You'd expect bigger matmuls to always be faster (more work to hide overhead) — but the curve is full of jagged cliffs where some sizes are dramatically slower than their neighbors. By the end of the lecture you'll understand exactly why.

To get there you also need a working mental model of the hardware. As Percy's first lecture stressed: scaling is about using resources effectively, and you can't be efficient without understanding the systems.

Why GPUs at all: the end of clock scaling

Compute is the currency of progress — faster hardware, better utilization, more chips, more parallelism all drive better models. In the 1990s we scaled clock speed (Dennard scaling: smaller transistors, faster clocks, serial instructions). That tapped out in the 2000s — transistor counts keep rising, but smaller transistors don't run faster (fundamental physics). So progress switched from going faster serially to scaling horizontally: many units executing in parallel. That's the GPU.

GPU FLOPs have grown super-exponentially (the Bill Dally curve): flat-ish through K20/M40, then takeoff at P100/V100. The hardware innovations behind it: tensor cores (V100, 2017), structured sparsity, and ever-lower number formats.

Inside a GPU: SMs and the memory hierarchy

The basic compute unit is the streaming multiprocessor (SM) — essentially a core, with internal streaming processors that run many threads in parallel. An A100 has ~128 SMs, each independently programmable, all sharing access to global memory. But the story modern hardware is really defined by is memory, not compute.

"Global memory" is what you normally think of as the GPU's memory (an H200's 144 GB). Shared/L1 and L2 sit physically on the chip — that proximity is why they're fast. Why not build the whole chip from shared memory? It's hundreds of times more expensive and power-hungry (SRAM must stay powered to hold values), so practical accelerators use a hierarchy you must respect. (Groq, recently acquired by NVIDIA, is an all-SRAM design that shines for inference.) Note: L1 and shared memory differ — the cache is automatic (stores recently-accessed data), while shared memory is programmable: you explicitly put things in and out.

The programming model: thread, block, warp

Thread (SIMT)

The lightweight unit of work. Crucially, GPUs are SIMT — all threads execute the same instructions on different inputs. Great for efficiency, constraining for programming.

Block

A group of threads guaranteed to run on a single SM — so a block shares that SM's shared memory. This is what makes tiling possible: blocks reuse the same memory.

Warp

The scheduling unit — 32 consecutive threads that execute together. The scheduler decides which warp runs next, swapping in lightweight threads to hide stalls.

The memory model mirrors the hierarchy: registers (per-thread, fastest), local memory (per-SM), shared memory (per-block — for data reused across threads), global DRAM (far, slow), plus constant and host (CPU) memory for offload. The single most important rule: the moment you go outside shared memory, things are slow. Grouping blocks to cut global-memory reads is the whole game.

A two-slide detour: TPUs (convergent evolution)

TPUs look surprisingly like GPUs — build an energy-efficient ML accelerator and you converge to the same place: a matrix-multiply unit, parallel vector units, a control system, and a fast/slow memory hierarchy (HBM + SMEM). TPUs are simpler: lighter control units, bigger matmul units. Every GPU concept maps onto a TPU one (both use systolic arrays for matmul).

The real difference is many-small vs few-big: an A100 has ~132 SMs / 528 matmul units; a TPU has ~2 units / 8 matmul units. Many small units give flexibility; few big units lock you into large matmuls (a TPU tensor core refuses inputs smaller than 64-dim). The networking differs most — but the core chip just lives to multiply matrices, so today's concepts transfer between GPU and TPU.

Two facts that shape everything

Matmul is the privileged operation. Before the V100 you programmed GPU shaders by hand to coax out matrix multiplies (genuinely clever hacks). Since the V100's tensor core does matmul in hardware, there's a 10×-plus throughput gap between matmul and any other floating-point operation. That's why every scalable ML architecture has a matmul at its core — it's the one way to get massive compute throughput.

Compute is outgrowing memory. Over time, total FLOPs grow fast; memory bandwidth grows slowly; interconnect (parallelism) bandwidth grows slowest. The compute-vs-memory gap keeps widening — which is exactly why almost every optimization below is a memory optimization. Inference is even more memory-bound, driving exotic moves like prefill/decode disaggregation (heavy-matmul prefill on one chip, memory-bound decode on another) and even per-layer disaggregation (attention vs MLP on different accelerators, e.g. Step-3).

The frame: the roofline model

Plot throughput against operational (arithmetic) intensity — compute done per byte moved. Below a threshold you're memory-bound (the diagonal: more compute doesn't help, you're waiting on bytes). Above it you're compute-bound (the flat roof: matmul units saturated). The goal is to live on the flat part — i.e. raise arithmetic intensity above the kink.

Six tricks follow. The first is GPU-specific; the rest are all about reducing memory movement.

Trick 1 · Avoid control divergence

Because threads are SIMT, an if statement behaves strangely. On a CPU you take one branch. On a GPU every thread executes both branches, masking out the side it isn't on — so the two branches run serially while half the threads sit idle. This is control divergence.

That's why GPU code avoids if: a ReLU is written as a multiply-by-mask, not a conditional, so all threads do the same multiply in one go.

Trick 2 · Low-precision computation

Half the bits = half the bytes to move. A ReLU over a vector reads one value and writes one: in fp32 that's 8 bytes per FLOP; in fp16, 4 bytes per FLOP — you've halved the memory traffic.

In practice a low-precision matmul downcasts the inputs, accumulates partial sums in full precision, and emits fp32. The "black art" is choosing which operation lives in which format — matmul inputs can be low precision, but softmax/exp usually need fp32. Getting low-precision training stable took years of incremental empirical work.

Below 16 bits there's no single standard. fp8 comes as E4M3 (more mantissa) or E5M2 (more range), used in different places. The newer block formats are wilder:

You only quantize layers that are safe (the first and especially the last layer are hard — the last drives a large share of the loss, so quantizing it causes instability and loss spikes). The payoff isn't a clean 2× — quantize/dequantize overhead means matmuls get ~20–30% faster. FP4 training is coming but not yet proven for big serious models in the wild.

Trick 3 · Operator fusion

Think of the GPU as a factory (compute) with a warehouse (memory) and a conveyor belt between them. Many small operations mean shuttling raw materials back and forth repeatedly — bandwidth you pay over and over. Fusion builds one big factory: read from global memory once, do all the work inside the SM, write back once.

# Unfused: sin²+cos² → each op reads & writes global memory
x → sin → square ↘
                  add → out      # 5 kernels, 5 round-trips to HBM
x → cos → square ↗

# Fused: one kernel reads x once, computes everything in-SM, writes once
out = fused(x)                          # 1 round-trip

Simple fusions like this are done automatically by torch.compile or the JAX compiler. (Advanced fusions sometimes need manual kernels — that's where FlashAttention comes in.)

Trick 4 · Recomputation (rematerialization)

Backprop normally stores the forward activations to reuse on the backward pass. Viewed as a systems problem (minimize memory, not just compute a gradient), you can instead throw them away and recompute on the fly in the backward pass — worth it when compute is abundant and memory is the bottleneck. For a chain x → sigmoid → sigmoid → sigmoid → out:

Same computation of the gradient, ⅝ the memory traffic — trade a little extra compute for a lot less memory movement.

Trick 5 · Memory coalescing

Global memory is DRAM, which delivers data in bursts: access one address and a whole contiguous block (~128 bytes) comes back essentially for free, because the memory cells are wired so an amplifier reads out a whole row/column at once. A read is coalesced when all 32 threads in a warp fall in the same burst.

This is why matrix layout (row- vs column-major) matters: certain memory-access patterns on a matmul are "privileged" and let you read big blocks in one shot.

Trick 6 · Tiling (the big one)

The most impactful idea: do as much as possible within the memory hierarchy. Cut the matrices into tiles, load a tile from global memory into shared memory once, then reuse it heavily before writing back.

Naively each input element is read N times from global memory. With tile size T it's read N/T times from global memory and T times from fast shared memory — a T-fold reduction in global access. PyTorch's matmul kernels tile for you; you'd hand-tile only for exotic ops.

Solving the mystery plot

Now the jagged curve makes sense — three effects stacked:

It's all just tiling + recomputation

FlashAttention turns PyTorch's naive multi-kernel attention into a single cleverly fused kernel, with dramatic latency and memory wins — the gains come from slashing data transferred to/from HBM. The paper says it does two things: tiling and recomputation, arranged so memory accesses are subquadratic. We now understand both.

Attention is three matmuls and a softmax: Q·Kᵀ → softmax → ·V. The matmuls tile straightforwardly (Figure 1 of the paper is literally tiled matmuls). The hard part is the softmax, a global operation that ties all the tiles together.

The key trick: online softmax

Normally softmax subtracts the max (for numerical stability), exponentiates, then normalizes — needing all values at once. The online version computes it incrementally: keep a running max and running sum of exponentials; whenever a new larger max appears, rescale the accumulator to correct for it; divide once at the end.

for tile in tiles:
    m_new = max(m, tile.max())          # track running max
    acc   = acc * exp(m - m_new)        # correct previous sum for the new max
    acc   = acc + exp(tile - m_new).sum()
    m     = m_new
out = acc_weighted_V / acc                # normalize once at the end

Because it's online, the softmax can be computed tile by tile — no tile needs to see the others. So the whole attention fits the tiling pattern:

Keep the small running statistics in shared memory/registers, fuse it all into one kernel (no round-trips), and on the backward pass don't store the N²-sized attention matrix — recompute it tile by tile. Tiling + online softmax + fusion + recomputation = FlashAttention. Every piece is a trick from Part 2.

Golden rules & takeaways

• GPUs trade latency for throughput: hundreds of lightweight SIMT cores, scheduled in 32-thread warps, blocks pinned to one SM's shared memory.
• Respect the memory hierarchy. Registers/shared/L1 are fast and on-chip; global HBM is ~10× slower. Going outside shared memory is the cost that dominates.
• Matmul is privileged (10×+ faster than other FP ops via tensor cores) — every scalable architecture leans on it; compute is outgrowing memory, so memory is the bottleneck.
• The six tricks: avoid control divergence; use low precision; fuse operators; recompute instead of storing; coalesce memory reads; and above all, tile to reuse shared memory.
• Sizes matter for non-obvious reasons — divisibility (coalescing) and wave quantization (tiles vs SM count) produce real cliffs. Don't cargo-cult "multiples of 32"; understand why.
• FlashAttention is the payoff: tiling + online softmax + fusion + recomputation, nothing more.

Next: Percy on more advanced systems — parallelism across many GPUs.

References

Primary source is the lecture itself. Resources and works it points to:

1. Horace He — Making Deep Learning Go Brrrr From First Principles
2. GPU Mode (formerly CUDA Mode) — community lectures & kernels
3. Austin et al. — How To Scale Your Model (TPU/GPU systems)
4. Dao et al. (2022) — FlashAttention
5. Dao (2023) — FlashAttention-2
6. Karpathy — nanoGPT (the vocab-padding speedup)