Why verifiable rewards change everything

The difference between RLHF and AlphaGo-style success is the reward. AlphaGo optimizes exactly what you want (win/loss, no sloppiness) — a search problem where more compute keeps helping. RLHF optimizes a learned, gameable reward — a learning problem that eventually overfits no matter how you regularize. Verifiable domains (a math answer is right or wrong, code passes tests or not) behave like the former: unhackable rewards let you scale compute and keep improving monotonically. The algorithms barely change from RLHF; where you end up is dramatically different.

PPO, and why people flee from it

Everything rests on the policy-gradient / REINFORCE identity: ∇ = E[∇log π · reward] — gradient descent on reward as reward-weighted SFT updates. To reuse rollouts (sampling is expensive) you go off-policy → TRPO (stay close) → PPO (a clipping heuristic). PPO's pseudocode looks simple, but there's a blog literally titled "the 37 implementation details of PPO" — a sign of an algorithm acutely sensitive to implementation. It needs a value model (as big as the policy → memory cost), has token-level KL, and people routinely degrade the GAE to γ=λ=1 (back to a bandit) or clip the KL at 0 (which ruins it). It works — labs have turnkey PPO — but it's finicky for from-scratch researchers.

DPO isn't the universal escape: it's specific to pairwise Bradley-Terry feedback, and math problems aren't inherently pairwise. PPO is the general hammer; the community wanted a simpler general hammer — which is GRPO.

GRPO: drop the value function

GRPO (from DeepSeekMath) keeps PPO's spirit but removes its worst part — the value function. You still need an advantage (to cut variance), so instead of a learned baseline, compute a z-score within a group: sample k rollouts per prompt and score each relative to the group's mean and std.

# For each prompt: sample k rollouts, get rewards r_1..r_k
A_i = (r_i − mean(r)) / std(r)          # advantage = z-score within the group
objective = clipped(A_i)  −  β·KL(π ‖ π_ref)
# Online case: ratio π/π_old = 1, so clipping vanishes → just advantage − KL

No value network → a clean one-page implementation (k rollouts → reward → normalize → KL → REINFORCE gradient, with a tiny ε on the std for all-equal rewards). It works: in DeepSeekMath, GRPO beats rejection fine-tuning. This simplicity is why nearly all open RLVR is built on it.

DeepSeek R1 — the clean recipe

R1 matched OpenAI o1 (long CoT, RL, strong math) with a reproducible recipe — far more impactful for researchers than an unrunnable PPO. The striking ablation: R1-Zero takes a base model and runs GRPO with just accuracy + format rewards (no SFT) → near o1. Clean enough that you'll replicate it in the assignment.

Kimi K1.5 — different path, same destination

Came out alongside R1, beat it, gets less attention. It uses a DPO-inspired derivation (assume analytic maximization → solve for the reward → match it with a squared loss) that, when you take the gradient, lands at essentially a group-mean baseline + KL — re-inventing GRPO's core by different means (nice evidence about which components matter).

• Data curriculum: exclude multiple-choice; best-of-k difficulty filtering — only train on problems at the edge of capability (drop ones the model already solves in 8 tries, and ones it never solves); remove mastered problems as it goes.
• Short CoTs on purpose: unlike GRPO's runaway length, Kimi adds a length penalty — shorten correct answers, but make incorrect ones only slightly shorter than average so the model can still recover (if you crush wrong-answer length to zero, it can never climb out of a weak domain). Long CoTs cost inference; thinking for 5 minutes beats an hour.
• Verification is hard: code uses generated test cases; math uses a reward model for answer-equivalence — ironic, since we started wanting "truly verifiable" rewards, but strict checkers fail constantly (equivalent answers, formatting drift). RLVR's "V" is a real rabbit hole.

RL infra is brutal (training + inference combined): a single hard rollout (think the Riemann hypothesis) blocks the whole batch; you must split train/inference machines and move weights between them; and on-policy GRPO is stable but low-utilization, while reusing rollouts (off-policy) destabilizes training. Kimi's ablations show RL beats expert iteration (training on correct answers) consistently.

Qwen 3 & Qwen3-Coder — the tried-and-tested playbook

Qwen 3's full pipeline: base → SFT → reasoning RL → thinking-mode fusion → RLHF → distill to smaller models. It uses the best of R1 + Kimi (difficulty filtering, decontamination, removing problems solvable without CoT) and does RL on just ~4,000 examples — the rest of the pipeline carries it. Thinking-mode fusion puts long-CoT and instant-response in one model toggled by a prompt tag; an early-exit trick makes performance degrade gracefully with the thinking budget, and thinking mode beats instant across budgets. (Later Qwen separated the modes again — the fusion quality drop was unacceptable.)

Qwen3-Coder-Next is the most detailed agentic-RLVR report: heavy mid-training (repo concatenation for long context, PR+RAG synthetic context, text+code markdown, agent traces, fill-in-the-middle), then it trains separate expert models (web dev, UX, QA, SWE agent) and distills them all back into one model — and builds GitHub agent environments at scale for RL, reaching ~70.6% SWE-bench with only a 3B-active-param model.

Golden rules & takeaways

• It's all about the reward. RLHF and RLVR share algorithms; the difference is reward hackability — unhackable verifiable rewards let you pour in compute and keep improving.
• Know GRPO cold (group z-score advantage + KL, no value function) — it's the workhorse of open RLVR and what you implement in the assignment. But remember it's not a clean policy gradient (length + std normalization have real side effects).
• R1's lesson: a base model + GRPO + accuracy/format rewards gets near o1; outcome supervision beats PRMs; and long-CoT reasoning distills.
• Manage CoT length (Kimi) and filter by difficulty (best-of-k) — train at the edge of capability, keep thinking short for inference cost.
• Verification and RL infra are the hard parts — answer-equivalence checking, long-rollout straggling, on- vs off-policy, and reward hacking.
• RLVR is finicky but no longer brutal like old-school PPO — smoother than you'd expect, and the pipelines (R1/Qwen) compose all the course's pieces.

This completes the from-scratch arc: tokenization → architecture → systems → scaling → data → alignment → reasoning.

References

Primary source is the lecture itself. Key works named:

1. Schulman et al. (2017) — PPO
2. Shao et al. (2024) — DeepSeekMath (GRPO)
3. DeepSeek-AI (2025) — DeepSeek-R1
4. Kimi Team (2025) — Kimi K1.5
5. Liu et al. (2025) — Understanding R1-Zero-like Training (Dr. GRPO)
6. Qwen Team (2025) — Qwen 3
7. Gao et al. (2022) — Reward Model Over-optimization
8. Zelikman et al. (2022) — STaR (expert iteration / self-taught reasoning)