Two philosophies, two case studies

Learning rate and batch size genuinely change with scale, so you face a choice: stabilize them (reparameterize so the optimum stays put) or predict them (fit a scaling law and extrapolate). Two papers embody the split:

MiniCPM — stabilize with μP, and the WSD schedule

MiniCPM trains a scaling ladder of much smaller models (~5× below the target) to nail the sensitive hyperparameters once. Two transferable ideas:

μP initialization makes the optimal LR scale-invariant by adjusting inits and per-parameter learning rates: scale the embedding output, scale residual connections by 1/√(layers), init matrices by the fan-in/fan-out ratio, scale per-tensor learning rates, and scale the LM head. In their experiments it works cleanly — LR sweeps across model sizes all minimize at ≈1e-2, so you don't have to retune the LR.

But even with a fixed LR optimum, the optimal batch size still varies with data and model size — and it follows a power law in the target loss (the Kaplan critical-batch-size pattern). So you can read it off once you know your loss target.

With WSD, a Chinchilla analysis is cheap (rewind + re-decay for data sweeps). MiniCPM replicated Chinchilla (methods 1 & 3) and got reasonable fits, though with somewhat different exponents (claiming more tokens-per-param — unclear if real or a fitting artifact).

DeepSeek LLM — fit the scaling law instead

Rather than stabilize, DeepSeek fits scaling laws for the optimal batch size and learning rate. They grid-search LR × batch at several scales, find the optimum at each, and fit lines vs non-embedding FLOPs: at higher FLOPs you want a bigger batch and a smaller LR. The batch-size fit is clean; the LR fit is noisier (grid quantization error can make it look arbitrarily linear). They use WSD (with two decay phases) and IsoFLOP sweeps to get clean Chinchilla curves — and their final predictions land close to the real models they trained. This replicates Chinchilla at large-ish open scale, confirming the analysis is sound.

The rest of the field (2025–26)

The core machinery (Chinchilla, LR scaling) is now standard, so recent papers say less about it and focus on what's new — mostly MoE:

Bottom line: stabilize (μP) or fit (scaling law) — both are validated at scale; the choice is taste plus compute budget.

What governs the optimal LR and batch size?

The StepFun hyperparameter study burned huge compute gridding LR × batch across models and data sizes. People don't even agree on which variables matter (Kaplan: batch ∝ terminal loss; DeepSeek: compute-based; StepFun: batch ∝ data) — so treat any single formula skeptically, but learn from the phenomena:

• The loss surface over (LR, batch) is smooth and roughly convex — so gridding for the optimum is viable.
• Optimal batch size depends mainly on total data D (a power law) — roughly batch ≈ √D · c, largely independent of model size.
• Optimal LR: smaller for bigger models, larger for more data (the data dependence is counterintuitive and contested/fragile).
• LR is fairly robust (the familiar 1e-3 … 1e-4 range is a good default) and the laws transfer to MoEs (controlling active params), but shift with training-data choice — so the exact numbers are contingent.

Under Chinchilla scaling (N and D both driven by compute), this reduces to the DeepSeek pattern: LR ↓ with compute, batch ↑ with compute.

Optimizers and the two scaling axes

On the NanoGPT speedrun, Muon beats Adam substantially and isn't much slower — but does it hold at scale? This is the hard, recurring research problem: a scale-dependent intervention looks great small, and you must decide about large. A careful optimizer-comparison study surfaces the pitfalls:

And establishing scaling is genuinely hard: a Marin run showed a beautiful trend for many orders of magnitude — then blew up past 1e20-ish FLOPs; the fix was a careful μP-style parameterization + optimizer change. Good-looking scaling can suddenly bite.

Muon, in brief

B[t] = mu·B[t-1] + grad            # momentum (as usual)
update = NewtonSchultz(B[t])         # orthogonalize: USVᵀ → UVᵀ (all singular values → 1)
param -= lr · update

Insight: not all parameters are alike — matrix params have a spectrum. Where Adam/AdaGrad normalize per-coordinate, Muon normalizes in the spectral norm (every direction unit-size). It only applies to matrix params (vectors like RMSNorm gains still use AdamW), and NewtonSchultz is a matmul-only approximation to orthogonalization (no slow SVD). Kimi K2 is fully trained with Muon (plus stability fixes) and is an excellent model — so Muon works at scale, even if "better than Adam at scale?" lacks an ablation. The meta-lesson: it's very hard to know what works at scale, so small-scale experiments + transfer remain how we do science.

The goal and the derivation

Goal: as width grows, keep the optimal learning rate fixed. Knobs: per-layer initialization, per-parameter learning rate, and residual scaling by model size. Validated at scale (Cerebras-GPT, MiniCPM) — μP fits are far more stable and land on the money. The derivation (Bernstein's review is the accessible one) is "physicist math" — track orders of magnitude under two invariants as width → ∞:

A1 — activations stay unit-scale at init

They shouldn't blow up or vanish with width (the norm of a layer's activation vector ~ √(hidden units)).

A2 — one gradient step changes activations by O(1) (feature learning)

The change must stay a fixed size with width — unlike the NTK regime, where it vanishes and the network stops learning features.

Working the operator-norm of a random Gaussian matrix (A1) and the rank-1 weight update's effect on activations (A2, assuming the loss update is O(1)) yields simple per-layer rules:

So under Adam, layers with large fan-in get smaller learning rates — layer-adaptive LRs that fall out of the scaling argument. The broader value is the method: assert invariants under a scaling limit, add assumptions, solve for the hyperparameter scalings.

Does it hold up? Stress tests

An independent replication confirms μP transfers LR optimality cleanly across scales, and is mostly robust to things the theory technically forbids (SwiGLU, RMSNorm, init variations). What breaks μP: a learned RMSNorm gain, sign-based optimizers (Lion), and — most concerning — large decoupled weight decay.

Golden rules & takeaways

• Two ways to handle the sensitive knobs (LR, batch): stabilize them with μP (MiniCPM) or fit a scaling law and extrapolate (DeepSeek). Both work at scale.
• WSD is the schedule to know: warmup → long stable → fast decay. Decay reclaims the gains, and rewind-and-re-decay makes data sweeps cheap.
• Optimal batch size tracks total data (~√D); LR is smaller for bigger models (and the ~1e-3–1e-4 range is a robust default). Under Chinchilla: LR↓, batch↑ with compute.
• Always check both scaling axes — compute and the Chinchilla (data:model) ratio — and tune your baseline (Adam) well before declaring a new optimizer better.
• Muon normalizes matrix updates in the spectral norm (via NewtonSchultz) and now trains real models (Kimi K2) — but small-scale wins don't guarantee large-scale wins.
• μP falls out of asserting activation invariants under a width limit; it's effective but breaks under learned norm gains, sign optimizers, and large weight decay.
• Scaling in the wild is an art — do reasonable things to maximize the chance of transfer; there's no guaranteed extrapolation.

Next: the data unit — evaluation, curation, filtering, deduplication, and mixing.

References

Primary source is the lecture itself. Key works named:

1. Hu et al. (2024) — MiniCPM (μP, WSD)
2. DeepSeek-AI (2024) — DeepSeek LLM (LR/batch scaling laws)
3. Hägele et al. (2024) — WSD / constant-LR cooldown analysis
4. StepFun (2025) — Predictable Scale (LR/batch hyperparameter scaling)
5. Liu et al. (2025) — Muon is Scalable for LLM Training (Kimi)
6. Yang et al. (2022) — Tensor Programs V (μP / hyperparameter transfer)
7. Dey et al. (2023) — Cerebras-GPT (μP at scale)
8. MiniMax (2025) — MiniMax-01 (architecture scaling)