Everything you didn't want to know about architectures

The method: learn from everyone's experiments

The best way to understand architecture is to train your own models and try variations — that's the course philosophy. But we lack the compute to sweep the whole design space, so the second-best move is to zoom out across many real models and read the pattern: the parameters everyone converges on are probably load-bearing; the ones that vary widely probably don't matter much.

A useful historical framing:

Where to put the LayerNorm — the one thing everyone agrees Vaswani got wrong

The original transformer puts LayerNorm inside the residual path (post-norm). Every modern model instead pushes the norm outside the residual stream. The canonical choice is pre-norm: normalize before each sub-layer (attention, FFN), leaving the residual stream itself untouched.

Why it stuck: keep the residual stream clean. With pre-norm, an unbroken path runs from input to output, so at initialization gradient magnitudes stay constant on the way back — simple signal propagation, better stability, and the ability to go deep. Post-norm puts a LayerNorm on the backward path at every block, rescaling gradients as they flow and producing larger, more frequent gradient spikes.

The original motivation was even narrower: pre-norm lets you remove learning-rate warmup. People found removing warmup hurt convergence badly under post-norm, but pre-norm converged cleanly. (Modern training still uses warmup anyway — but the deeper stability/depth benefits are why pre-norm won.)

RMSNorm: a free systems win

LayerNorm subtracts the mean, divides by the standard deviation, then scales and adds a bias. RMSNorm drops the mean-subtraction and the bias — it only rescales by the root-mean-square:

# LayerNorm: center, normalize, scale, shift
y = (x - mean(x)) / std(x) * gamma + beta

# RMSNorm: just normalize by RMS, then scale
y = x / sqrt(mean(x**2) + eps) * gamma

LayerNorm is strictly more expressive, but in practice there's no measurable quality loss — and RMSNorm is faster. This is the first taste of architecture/systems co-design: the win isn't FLOPs, it's data movement. A norm is only ~0.17% of total FLOPs but can be up to ~25% of runtime on small models, because it's a low-arithmetic-intensity operation that shuffles activations between fast and slow memory without doing much math. Strip the parts that move memory without buying expressiveness.

Narang et al. (2020) confirmed it: on a 200M-param transformer, switching to RMSNorm gives more steps/second and slightly better loss. The same logic explains why modern models drop bias terms on linear layers too — low arithmetic intensity, little benefit, and biases can even cause stability issues. So nearly everyone uses RMSNorm with no biases.

Activations: gating is what matters

There's a zoo — ReLU, GeLU, Swish, ELU, SeLU, and the gated variants GeGLU, SwiGLU, ReGLU, LiGLU. You can train a perfectly good model on a plain activation (Chinchilla on something simple; GPT-3 on GeLU, which is just ReLU with a small differentiable "divot" near zero). But the real action is in gated linear units (GLUs) — almost every credible modern model uses one.

A standard FFN versus a gated one:

# Plain ReLU FFN: two matrices
ffn(x) = relu(x @ W1) @ W2

# Gated (ReGLU): a third matrix V gates the activation entrywise
ffn(x) = (relu(x @ W1) * (x @ V)) @ W2
#          \____activation____/   \_gate_/

Name the GLU by its activation: ReGLU (ReLU), GeGLU (GeLU), SwiGLU (Swish/SiLU = x·sigmoid(x)). The gate x·V modulates the activation before the down-projection.

Lineage: Google models (T5, Gemma) use GeGLU; PaLM and the whole Llama family use SwiGLU (the more dominant one, though among GLUs the choice barely matters). Non-GLU survivors exist — GPT-3 (GeLU), Nemotron-340B (squared ReLU) — but they're rare. Narang et al. (2020) again confirm GLUs win on loss and downstream metrics.

Parallel vs serial layers — a fun idea that faded

Normally a block runs attention then MLP (serial), which creates a dependency. The parallel block (from GPT-J, popularized by PaLM) instead computes both from the same input and adds both into the residual stream at once:

# Serial (standard):  x = x + mlp(norm(x + attn(norm(x))))
# Parallel (GPT-J/PaLM): x = x + attn(norm(x)) + mlp(norm(x))

Done right, you can share LayerNorms and fuse the matmuls — a real systems win (PaLM reported ~15% utilization gain, no quality drop). But it has fallen out of favor: serial-form optimization got good enough that the systems gain no longer justifies the representational hit (you effectively lose half your depth). Cohere (Google-influenced) still uses it; most don't, and the evidence is genuinely mixed because clean ablations are scarce.

Position embeddings: the still-in-flux part

Attention is just inner products — it's permutation-invariant, so you must inject position somehow. The history:

Sinusoidal (original transformer)

Add sines/cosines to the token embeddings; position is recoverable via Fourier intuition.

Absolute (learned)

Each position index gets its own learned embedding, added to the token vector.

Relative (T5, Chinchilla)

Don't touch the token vectors — add a learned offset directly to the attention matrix based on the distance between positions.

RoPE — Rotary Position Embedding (dominant post-2024)

A relative scheme that preserves inner-product structure. Originated in a GPT-J-era blog/paper; now nearly universal.

RoPE's design goal: the score between two tokens should depend only on their relative offset, not their absolute positions. Sinusoidal fails this (absolute cross-terms leak through); absolute obviously fails; "relative" embeddings satisfy it but aren't true embeddings (no inner-product structure, just additions to the attention matrix). RoPE achieves it with a beautiful trick: inner products are invariant to rotation, so rotate each token's vector by an angle proportional to its position.

In the example, "we know" appears at positions (0,1) in one sentence and (2,3) in another — the absolute angles differ, but the relative angle between the two words is identical, so their attention score is unchanged. In D dimensions you do the simplest thing that works: chop the vector into pairs of coordinates and rotate each pair, with different frequencies per pair — slow rotations capture long-range relationships, fast rotations capture "are these neighbors?". Apply it at the attention level to the queries and keys (multiply by sines/cosines), not to the bottom embeddings, so position-invariance is enforced at every attention computation. (Gemma 4's "p-RoPE" rotates only the first coordinates — a tweak for tiny models.)

Hyperparameters: a small, forgiving space

Confronting a model for the first time, the hyperparameter space looks daunting. But the space people actually use is small, and most knobs are forgiving.

Feed-forward ratio (ff_dim ÷ d_model)

Rule of thumb: 4× for non-GLU models. With a GLU and the ⅔ correction you land near 2.67; Llama bumped it to ~3.5 (citing efficient attention, emphasizing the MLP a bit more). The wild exception is T5 at 64× (a systems argument: bigger matmuls keep hardware busier) — and tellingly, T5 v1.1 reverted to ~2.5. Kaplan's sweep shows a flat basin from ~1 to ~10; loss only shoots up (quadratically) past ~100. So any value in the 2.6–4 range is safe.

Head dimension

Convention: (#heads) × (head_dim) = d_model — i.e. ratio ≈ 1, so multi-head costs the same as single-head. Forgiving, with a wide basin (T5 and LaMDA are exceptions).

Aspect ratio (d_model ÷ n_layers) — the one to actually think about

Most models cluster around ~100 (width per layer): GPT-3, Llama, and others. This trades expressiveness against hardware: very deep models force pipeline parallelism (painful), while width is easy to split via tensor parallelism. Kaplan again shows the optimum is similar across model sizes and fairly flat — and EleutherAI-style studies found that across depth/width trades, FLOPs mostly determine quality, not the aspect ratio. So pick ~100 and worry about systems utilization instead.

Vocabulary size

Two clear classes: early monolingual (English-only) models used ~30k; modern multilingual/production models (and bigger models generally) use 100k–200k. Google models trend largest; Llama derivatives ~100k. Multimodal systems add a separate, often large image-tokenizer vocab. Bigger models can support bigger vocabularies (scaling-law result).

Regularization — counterintuitive

In compute-constrained LM training you do roughly a single pass over more data than you can process, so overfitting essentially never happens — some teams only watch training loss. Yet weight decay remains common (dropout less so). Why? Because weight decay acts as an optimization intervention, not a regularizer: combined with learning-rate decay, stronger weight decay starts slower but converges to a meaningfully better minimum. It doesn't shift the train/val gap (there's no overfitting to fix) — it changes the optimization trajectory. A clean reminder that these knobs interact in non-obvious ways.

Stability tricks: taming the softmaxes

As runs get expensive, stability beats marginal performance — a single blow-up can waste millions of dollars and leave an unrecoverable model. The usual suspect is the softmax: it has an exponential (blows up fast) and a division (dangerous). There are two softmaxes — the output (vocabulary) softmax and the attention softmax — and both are danger zones.

Output softmax → z-loss

The loss is log p = u − log Z. The model output u is usually well-behaved, but the normalizer Z = Σ exp(·) can explode or collapse to 0. The softmax is over-parametrized (adding a constant to all logits cancels out), so you can freely push log Z toward 0 with a penalty:

loss += lambda * (log Z)**2   # keep the normalizer near 1 → numerically stable

This z-loss (Devlin, 2014; revived by Baichuan, DCLM, OLMo) is a surprisingly effective output stabilizer.

Attention softmax → QK norm

Apply a norm to the queries and keys before their matmul, so the inputs to the attention softmax have roughly unit scale and can't run away:

q, k, v = project(norm(x))
q, k = rms_norm(q), rms_norm(k)   # QK norm — keeps softmax inputs ~unit scale
attn = softmax(q @ k.T / sqrt(d)) @ v

QK norm came from the multimodal world (Idefics, Chameleon) and is now standard. It doesn't hurt quality and prevents attention degeneracies — and even lets you push the learning rate higher. (It's the same "sprinkle in LayerNorms" lesson: first pre-norm, then post-sub-layer norms, now norms on Q and K.)

Logit soft-capping (a stronger, Google-specific lever)

Instead of controlling the inputs, hard-bound the outputs: squash logits through a tanh so they can never exceed a cap (Gemma 2/3/4). It's a near-hard constraint. NVIDIA's systematic comparison found QK norm helps (enables higher LR), but soft-capping alone can lose performance — capping means you can never express a very confident prediction. Safe, but not free.

Attention variants (dense only — SSMs next time)

Why GQA exists: the KV-cache problem

During training/prefill you process the whole sequence at once, so attention is a big matmul with good arithmetic intensity (compute-bound — GPUs happy). But at serving time you generate one token at a time (autoregressive), caching past keys/values in a KV cache. Each decode step must re-read parameters and the growing cache, so arithmetic intensity collapses to roughly 1 / (n/d + 1/b) — the n/d term (sequence length over hidden dim) is hard to reduce and leaves you memory-bound.

MQA (multi-query): share a single key/value across all query heads — the KV cache shrinks dramatically, raising arithmetic intensity by a factor of the head count, but it costs real expressiveness. GQA (grouped-query): keep all the query heads but use a small number of groups of K/V heads — a tunable dial between MHA and MQA. The trade-off is unusually favorable: a small reduction in K/V heads keeps almost all the quality while slashing inference cost, which is why nearly all current models adopt GQA. (Note: this is trained-in, not an inference-only swap. DeepSeek V2's multi-head latent attention (MLA) is a different factorization — next lecture.)

Sliding-window / sparse attention for long context

An old idea (GPT-3 alternated full attention with banded/local windows) now hugely popular for long context. The pattern: interleave a few full-attention layers with many cheaper local sliding-window layers — e.g. Cohere Command A uses full attention every 4th layer, sliding windows in between; stacking lets local layers eventually aggregate global information. Some models drop position embeddings (NoPE) on the long-range layers. Llama 4, Gemma 4, OLMo 3 all combine sliding-window + full attention; Qwen3.5 instead alternates a gated-DeltaNet state-space layer with full attention every 4 layers.

The pattern across models

Reading the modern dense models together, the consensus is striking — and the genuine variation is concentrated in position embeddings and long-context handling.

Golden rules & takeaways

• Keep the residual stream clean. Push norms outside it (pre-norm) for stable gradients and depth; sprinkle extra norms when stability wobbles.
• Prefer RMSNorm and drop biases. The win is data movement, not FLOPs — strip low-arithmetic-intensity ops that don't buy expressiveness.
• Use a gated activation (SwiGLU/GeGLU) with the ⅔ feed-forward correction — a consistent, near-free gain.
• RoPE for position — relative, rotation-based, applied to Q/K at every attention layer.
• Most hyperparameters are forgiving (ff-ratio ~4 or ~2.67, head-ratio ~1, aspect ratio ~100). Optimize for systems utilization, not for squeezing the basin.
• Weight decay is an optimization tool, not a regularizer in single-pass LM training — it interacts with LR decay to reach a better minimum.
• Stabilize the softmaxes: z-loss on the output, QK norm on attention; soft-capping is a stronger but costlier option.
• GQA is the default attention for cheap inference at near-full quality; long context is going hybrid (full + sliding-window / SSM).

Next lecture: mixture-of-experts (MoE) and state-space models / linear attention.

References

Primary source is the lecture itself. Key works named:

1. Vaswani et al. (2017) — Attention Is All You Need
2. Xiong et al. (2020) — On Layer Normalization in the Transformer Architecture (pre- vs post-norm)
3. Zhang & Sennrich (2019) — Root Mean Square Layer Normalization (RMSNorm)
4. Narang et al. (2021) — Do Transformer Modifications Transfer Across Implementations and Applications?
5. Shazeer (2020) — GLU Variants Improve Transformer (SwiGLU/GeGLU)
6. Chowdhery et al. (2022) — PaLM (parallel layers, z-loss)
7. Su et al. (2021) — RoFormer: Rotary Position Embedding (RoPE)
8. Kaplan et al. (2020) — Scaling Laws for Neural Language Models (ff-ratio, aspect-ratio sweeps)
9. Shazeer (2019) — Fast Transformer Decoding (Multi-Query Attention)
10. Ainslie et al. (2023) — GQA: Training Generalized Multi-Query Transformers
11. Child et al. (2019) — Generating Long Sequences with Sparse Transformers (sliding window)