The setup: pre-training is the base, post-training extracts behavior

Pre-training scales broadly and is the foundation — you can't train your way to a good assistant without it. But to pull the behaviors you want out of that "primordial soup," you do explicit data collection and steering: messy, engineering-heavy work. A caveat up front: frontier post-training info is sparse (trade secret). Good references are old (the RLHF "learning to summarize" paper, Anthropic's HH, with detailed annotation guidelines), and open recipes mostly rely on distillation — different from the frontier's human data collection. The recipe is two parts: collect demonstrations → SFT; collect preferences → RLHF.

The data progression

SFT's method is just pre-training (next-token prediction) — only the data differs. The evolution of that data tells the story:

Three high-level shifts: toward chattiness (human-like vs programmatic NLP outputs), toward higher-quality/expert annotators, and toward tool use. A key reversal from FLAN: with a strong enough base model, you need few high-quality examples, not massive scale — pre-training generalization carries you.

SFT pitfalls: style, knowledge, and safety

Knowledge & hallucination. An SFT example like "…(Reference: Bivens & Michel, 2013)" teaches two things at once: the fact and the format ("give a reference"). If you train on tail knowledge the model doesn't already have, it learns to emit references/facts it can't actually produce → hallucination. So you may not want the highest-quality data if the model doesn't know it. (Schulman's argument: this is why you need RL — "knowing what you know" is policy-dependent; an external teacher shoving facts in can't calibrate the model. RL can surface an internal "I know this" signal by rewarding references only when they're right.)

Safety SFT. Balance the violation rate (bad queries getting through) against the false-refusal rate ("how do I kill a Python process?" → refusal is annoying). It takes surprisingly little data — a few thousand examples (Llama 2), ~50K (OLMo), often mined from real usage (WildChat jailbreaks → preferred refusals). With as few as 500 examples the unsafe-response rate drops dramatically — the model already has a "safe/unsafe" direction after pre-training; SFT just surfaces it. (More examples still help for fine-grained distinctions.)

SFT is bleeding into pre-training

The method is boring (gradient descent), but one important trend: instruction/chat data is increasingly mixed into the pre-training decay phase ("mid-training") — the closest-to-deployment, lowest-LR part, ideal for high-quality data. This scales up instruction tuning and blurs the line so much that a "base model" today is partly a lie (it's seen UltraChat and friends). A practical workflow: run cheap data ablations during the decay phase, then reflect those quality estimates back into the pre-training mix (you can't make all of pre-training high-quality — you'd run out of tokens).

The conceptual shift, and why RL at all

Why bother? Two reasons: (1) people's stated vs revealed preferences differ — freelance writers sometimes preferred Instruct-Davinci's summaries over their own writing, so it's better to rate outputs than only collect demonstrations; (2) verification is often easier than generation (checking a proof vs writing one). The RLHF loop:

The data and the annotators

Preference data is collected as pairwise ratings. InstructGPT rated for helpful, truthful, harmless; Bard used a Likert scale — both fairly short instruction sets. The annotator pool has shifted upward: bachelor's/master's holders, modal age ~35, $50–100+/hour for bespoke expert work (doctors, lawyers) as labs target white-collar deployment — while a large low-paid tier still exists (a bifurcated pyramid).

Model-based annotation has largely won (for catching up to the frontier): GPT-4 annotations agree with careful human ones at ~10× lower cost. Hugging Face's Zephyr tried hard to use only human data and ended up using AI feedback anyway; UltraChat/UltraFeedback/Tulu 3 are now standard. You still need humans to push the frontier (lawyers, scientists, world knowledge) — but models can bootstrap their own data (Constitutional AI for safety, Self-Instruct for capability). Caveat: models share human biases — you can RLHF on length alone and climb the benchmarks.

Algorithms: PPO → DPO

The objective (straight from InstructGPT): maximize reward under the policy, minus a KL penalty keeping it close to the reference (pre-trained) model so it doesn't go degenerate:

maximize  E_{y ~ π}[ r(x, y) ]  −  β · KL( π ‖ π_ref )

The classic optimizer is PPO, reached by a progression:

Policy gradient looks like SFT with reward-weighted examples, but it needs a fresh sample at every step (expensive). Off-policy reuses a rollout; TRPO keeps you from straying too far (via importance weighting); PPO replaces the hard distance constraint with a simpler clipping heuristic. (Full treatment next lecture.)

# per preference pair (y_win ≻ y_lose): gradient ∝
#   [↑ log π(y_win)  −  ↓ log π(y_lose)]  ·  (how wrong the implied reward is)

DPO is far simpler than PPO (just gradients) and works well enough for Llama (DPO + an outer rejection-sampling loop). Variants (SimPO, length-normalized DPO) barely differ, and DPO-vs-PPO results are contingent on the experimental setup (AI2 has papers concluding each way). Takeaway: "step toward good, step away from bad, set the step sizes right" works.

RLHF pitfalls

• Over-optimization. Push RLHF too hard and you overfit the learned reward model — the KL regularizer is critical to prevent it. You can't simply collect enough thumbs-up to reach superintelligence.
• Model collapse / lost diversity. A policy (unlike a distribution) can concentrate on a few outputs — RLHF models are notably less diverse.
• Lost calibration. Models are well-calibrated after pre-training but uncalibrated after RLHF (a rare open problem OpenAI flagged for GPT-4) — which matters a lot for the exploration that reasoning/RLVR needs.

This motivates the next lecture: are there rewards you can't over-optimize — where dumping compute keeps monotonically improving the model? That's why RLVR (RL with verifiable rewards) has been so impactful.

Golden rules & takeaways

• Post-training extracts behavior from a pre-trained base — it's data-heavy and artisanal, and frontier details are secret.
• SFT is all about the data: prefer few high-quality, natural examples; mind that style ≠ capability; and don't SFT on knowledge the model lacks (it causes hallucination).
• A little data steers a lot (≈500 examples for safety) — the behavior is already latent after pre-training.
• SFT data is merging into pre-training's decay phase, blurring "base" vs "instruct."
• RLHF maximizes a reward, not a distribution — useful because rating beats demonstrating and verifying beats generating; annotators (and their biases) shape the model; model-based feedback has largely won for catching up.
• PPO → DPO: DPO drops the reward model and rollouts (closed-form from a non-parametric assumption) and works about as well.
• Watch over-optimization, collapse, and calibration — the reward model is gameable; KL regularization is essential.

Next: RLVR and reasoning models — from ChatGPT to o1.

References

Primary source is the lecture itself. Key works named:

1. Ouyang et al. (2022) — InstructGPT · Stiennon et al. (2020) — Learning to Summarize from Human Feedback
2. Bai et al. (2022) — Anthropic HH (RLHF)
3. Wei et al. (2021) — FLAN · Wang et al. (2022) — Self-Instruct · Alpaca
4. Köpf et al. (2023) — OpenAssistant · Lambert et al. (2024) — Tulu 3
5. Rafailov et al. (2023) — DPO · Schulman et al. (2017) — PPO
6. Bai et al. (2022) — Constitutional AI · Tunstall et al. (2023) — Zephyr
7. Santurkar et al. (2023) — Opinions of LMs · Hosking et al. (2023) — Human Feedback is Not Gold Standard
8. Gao et al. (2022) — Scaling Laws for Reward Model Over-optimization