The scaling-law paradigm

At the frontier you can't just copy others' architecture choices and beat the state of the art — you have to optimize. Doing that by trial-and-error on huge runs is wasteful, so the move is to find simple, predictive rules connecting small-scale behavior to large-scale behavior. If small and large scales are linked by robust regularities, you can do all your optimization cheaply and extrapolate with confidence. To some labs this is "almost a way of life" — but as we'll see, scaling laws are subtle, engineered objects, not magic.

Power-law relationships hold for a remarkable range of factors. The reliable ones in pre-training put a resource on a log x-axis (compute, data, or parameters) and log test loss on the y-axis — and the result is a line. More exotic versions exist (downstream benchmarks → sigmoid; date-on-x → roughly linear capability envelopes), but those are less guaranteed. LM performance is just far more regular in scale than you'd expect.

A line on a log-log plot = a power law

Fix the model (bigger than the data), grow the data, watch the error fall toward an irreducible floor (the task's entropy). In the power-law regime (far from that asymptote), log loss vs log data is a straight line — "scale-free" — meaning error decays polynomially.

Where do the exponents come from? Two statistics warm-ups:

# Mean estimation: error = σ²/n  →  log error = −1·log n + C  →  slope −1
# Nonparametric (smooth fn in D dims): error ≈ n^(−1/D)  →  slope −1/D

Neural exponents are small (≈ −0.1 to −0.3) — much slower than the −1 of mean estimation. The mental model: a neural net learns from data roughly like a nonparametric regressor in ~10 dimensions (some argue the exponent literally tells you this — interesting, take with salt). The polynomial form is natural; the slowness is the surprise.

Slopes vs intercepts — the recurring lesson

Data scaling alone only forecasts "how fast do I learn." The engineering questions are about composition: what data mixture? repeat data or not? filter how hard? A theme emerges that you'll meet over and over:

• Repetition: "Scaling Data-Constrained LMs" found up to ~4 epochs costs you nothing; beyond that, realized scaling falls below the fresh-data projection. Even pushing to infinite compute (ensembling, regularizing) mostly moves the intercept — slopes stay similar.
• Filtering is scale-dependent: with little compute you filter aggressively (keep only the best); with lots of compute you loosen the filter and train on lower-quality data rather than over-repeat. Optimal filters aren't fixed across scale.

Architectures and optimizers

Want to know if an LSTM (or SSM, or SGD) beats the transformer (or Adam) at scale? Don't train one giant model — train many small ones across compute ranges and compare the lines. A worse slope is disqualifying (it only gets worse as you scale); a worse intercept usually is too. Every architecture paper (Mamba, gated DeltaNet) shows exactly this plot — "our model on top of, or below, the vanilla transformer."

That last point is why scaling is treated almost paradigmatically: "if it doesn't show up in the scaling law, it's not a good intervention." Past sweeps at tiny compute already predicted what frontier models adopt.

Scale-invariant hyperparameters & "what is a parameter?"

Some hyperparameters are best studied as scale-invariant quantities. The number of layers isn't (bigger models want more), but the aspect ratio (≈100 × d_model per layer) is — its optimum barely moves across model sizes (drifting slightly smaller for deeper models). So "fix the aspect ratio and scale up" is a defensible recipe you can verify with a plot.

For MoE, where total and active parameters decouple, there are clean joint scaling laws (Apple/MIT): minimizing loss at fixed compute pushes toward sparser models, and the non-active ("empty") parameters still reduce loss. All IsoFLOP-style.

The two you must re-tune: batch size & learning rate

You'll copy most choices, but batch size and learning rate have to be re-derived (and they're coupled). Batch size matters for systems too — bigger enables data parallelism — so the question is "how big before it hurts?"

Critical batch size

Up to a point, each extra batch element cuts gradient variance → perfect scaling (noise-limited). Past the gradient's noise scale you're bias-limited (your local descent direction simply disagrees with the global optimum), so returns diminish. The crossover — the critical batch size — is the biggest batch you can use without big efficiency loss.

# Pick a target loss; sweep batch sizes; record steps S and examples E (= S·batch).
# McCandlish et al.: balance the two → B_crit = E_min / S_min
# Estimable from gradient covariance / squared gradient norm.

Why it's in a scaling lecture: critical batch size grows (as a power law) as the loss improves — near the minimum, variance reduction is more valuable, so big runs can use big batches.

Learning rate

Under width scaling, bigger models want a smaller LR (rule of thumb ≈ 1/width — more parameters changing at once, so move less). Two philosophies, both used successfully at scale: (a) predict how the LR optimum shifts (a scaling law), or (b) reparameterize the model (init + per-part step sizes) to keep the LR optimum fixed across scales — μP and relatives. (Detailed next time.)

Upstream ≠ downstream

Perplexity is beautifully clean and predictable; downstream task accuracy is not. In Tay's data, the best-perplexity model (NL12) was not the best downstream model (NL32XL). So: fit scaling laws on perplexity (low variance — usually single runs suffice, since one-pass SGD makes train ≈ val), but be cautious about transfer to downstream. Pre-training teams who hand off "good perplexity" often hand off problems too.

The honest procedure: train many small models, fit lines on log-log plots, and if the fit is clean, trust the gap to persist and deploy. You should be able to predict your big run's numbers fairly precisely beforehand — that's the whole point.

More data or a bigger model?

Compute ≈ data × parameters. A tiny model fed endless data plateaus (wasted compute); a huge model on too little data is also wasteful. Fit a joint scaling law (Kaplan; Rosenfeld) — roughly a sum of a model-size term and a data term plus an irreducible floor — on a cheap low-compute corner, extrapolate (it fits remarkably well), then minimize loss subject to the FLOP budget.

Chinchilla's three methods

Method 1 — lower envelope

For each FLOP value, take the best loss achieved by any run; fit the model sizes at those envelope points. (→ 67B for their budget.)

Method 2 — IsoFLOP (the robust favorite)

Fix several FLOP budgets; for each, sweep the param↔data trade-off (double data, halve model…); fit a quadratic to each budget's losses and connect the minima. (→ 63B.)

Method 3 — parametric fit

Fit the hypothesized joint functional form to many runs by curve-fitting. (Sensitive to how you fit.)

Methods 1 & 2 agree (≈0.5/0.5 for data/params); method 3 differed (0.46/0.54). Kaplan's exponents were very different — hence their giant models.

Why Kaplan and Chinchilla disagreed

Neither did anything obviously crazy — yet predictions diverged. Porian/Yair's "Resolving Discrepancies" reproduces Kaplan, then closes the gap with a chain of minor-looking fixes:

You probably don't want the Chinchilla ratio

Chinchilla minimizes training compute — but at frontier labs most compute goes to R&D and serving, not the final run. For serving you want a small, capable model, so you deliberately "overtrain" (really, the right amount of training) far past 20 tokens/param.

Chinchilla matters less for its 20:1 number than for what it teaches about how to fit scaling laws carefully. And IsoFLOP endures as the go-to research tool: fix a FLOP budget, sweep the other degrees of freedom, read off the surface.

Golden rules & takeaways

• A line on a log-log plot is a power law. Loss decays polynomially in data / params / compute; neural exponents are small (slow, ~nonparametric-in-~10-D learning).
• Interventions move the intercept; the slope rarely changes. So the best small-scale choice is usually the best large-scale choice — which is why you can optimize cheaply.
• "If it doesn't show in the scaling law, it's not a good intervention." Compare architectures/optimizers by their lines, not single big runs.
• Re-tune batch size and learning rate. Use the critical batch size (it grows with compute); set LR by predicting its shift or by reparameterizing (μP).
• Fit on perplexity, be cautious about downstream transfer.
• Scaling laws are engineered lower bounds. Kaplan vs Chinchilla shows tiny choices (parameter counting, warmup, batch size) flip the conclusion — scale a recipe you'd actually want to scale.
• Don't chase the 20:1 ratio for production — overtrain small models for cheap serving. IsoFLOP is the durable research tool.

Next: inference (then a return to advanced scaling — μP and optimizers).

References

Primary source is the lecture itself. Key works named:

1. Hestness et al. (2017) — Deep Learning Scaling is Predictable, Empirically
2. Kaplan et al. (2020) — Scaling Laws for Neural Language Models
3. Hoffmann et al. (2022) — Training Compute-Optimal LLMs (Chinchilla)
4. Rosenfeld et al. (2019) — A Constructive Prediction of Generalization Error
5. McCandlish et al. (2018) — An Empirical Model of Large-Batch Training (critical batch size)
6. Muennighoff et al. (2023) — Scaling Data-Constrained Language Models
7. Porian et al. (2024) — Resolving Discrepancies in Compute-Optimal Scaling
8. Tay et al. (2021) — Scale Efficiently: architecture & scaling
9. Yang et al. (2022) — Tensor Programs V (μP / hyperparameter transfer)