CS336 · Lecture 12 — Evaluation

Four lenses on "good"

There's no single right answer — just different framings, each with blind spots:

A model is good if…	Source
it scores high on benchmarks	Artificial Analysis "intelligence index"
it scores high and is cheap to run	intelligence vs inference-cost (correlated, not aligned)
people prefer its responses	Arena AI (formerly Chatbot Arena)
people choose to pay to use it	OpenRouter usage rankings (economic lens)

One popular composite "intelligence" ranking. Source: Artificial Analysis / CS336 Lecture 12.

The most natural metric — and its limits

A language model is a distribution p(x), so the natural metric is perplexity: how much probability mass it assigns a test set D (normalized by length). It's exactly what training minimizes, so you measure it on a held-out split. This was the dominant paradigm through the 2010s (Penn Treebank, WikiText-103, the One Billion Word benchmark) — in-distribution: train split → test split. GPT-2 changed the game by evaluating zero-shot, out-of-distribution on those datasets, beating SOTA on small ones (e.g. PTB) it never trained on.

"Perplexity is all you need" (more faith than science)

If the true distribution is t and your model is p, the best possible perplexity is the entropy H(t), reached only when p = t. And if p = t, you can solve everything via p(solution | problem) — so "push perplexity down and you reach AGI." This belief drove scaling when the payoff wasn't yet obvious.

But perplexity is also more than you need: in "Stanford was founded in 1885" it charges bits for every token (the boring "founded" as much as the informative "1885"). The fix is conditional perplexity p(response | prompt), focusing on the tokens you care about. And several "accuracy" benchmarks are really perplexity in disguise — LAMBADA (cloze, fill-the-blank requiring long-range context) and HellaSwag (multiple-choice sentence completion).

⚠️ Perplexity leaderboards require trust

A submitter hands you a log-prob; you must trust the probabilities are valid (sum to 1) — otherwise an "LM" that always returns probability 1 wins. Downstream tasks avoid this: a black-box LM maps prompt → response and you score the response. Perplexity remains heavily used (it varies smoothly with scale, ideal for scaling laws), but believers aside, you need real-world benchmarks too.

Exam benchmarks — and the saturation treadmill

Exams give control over subject and difficulty with unambiguous, gradeable answers. The recurring pattern: a hard new benchmark saturates as models improve, prompting a harder one.

Benchmark	Idea	Trajectory
MMLU (2020)	57 subjects, multiple-choice; really tests knowledge, few-shot	GPT-3 above chance → now ~90+ (saturated)
MMLU-Pro	denoised, 4→10 choices, chain-of-thought	dropped to ~33 → back to ~90
GPQA	PhD-written, "Google-proof"; experts 65%, non-experts 34% w/ Google	GPT-4 39% → ~94
Humanity's Last Exam	multimodal, many subjects, private holdout	single digits → ~65 (still room)

Multiple choice survives because it can be made arbitrarily hard (its real limit is the kinds of questions, not difficulty). But exams don't capture real usage — nobody asks their assistant HLE questions. And every benchmark carries a contamination caveat: even without literal test-set training, questions derive from sources that are in the training data.

Chat benchmarks — scoring open-ended responses

Most real prompts are open-ended with no ground truth ("what herbs go with a beet + goat-cheese salad?"). Three approaches:

Chatbot Arena (humans, pairwise): Random users get two anonymized responses and pick the better; fit ELO ratings to the pairwise comparisons. Pros: real prompts, dynamic, no need to feed all models the same prompts (ELO only needs a connected graph). Cons: who are these users? binary preference conflates style and correctness; prone to sycophancy; how can a user judge correctness of an answer they sought?
AlpacaEval (LLM-as-judge): Win rate vs a baseline (GPT-4) judged by an LLM. Bias: judges favor long responses → leaderboard gaming → fixed by a debiasing regression. It correlates ~0.98 with Chatbot Arena (one way to "evaluate the metric").
WildBench (rubrics): Real human-chatbot prompts, LLM-as-judge with a per-task checklist/rubric — which scopes the ill-defined "is this good?" into something well-defined.

Lessons for open-ended eval

Pairwise comparisons of similar responses give higher signal than absolute 1–10 scores; beware judge biases (human and LLM); use multiple judges; and a rubric/checklist improves reliability regardless of who judges. The problem is genuinely not well-defined.

Agentic benchmarks — evaluating what models do

An agent = language model + scaffold (logic for tool use and control flow). These evaluate multi-step, tool-using behavior:

Benchmark	Task / grading
SWE-bench	given a repo + GitHub issue, submit a PR; graded by unit tests (16% → 93%; "Verified" subset fixes bad tasks)
Terminal-Bench	tasks in a computer terminal (simple, universal); crowdsourced; tasks take hours-to-weeks for humans
CyBench	40 capture-the-flag security tasks (hack a server, extract a flag); first-solve time as difficulty
MLEBench	Kaggle competitions (process data, train models, submit)

Scaffolds matter — a lot

Same model + different scaffold = different accuracy. Effective scaffolds use explicit planning (a to-do list), hierarchical delegation (sub-agents with clean context), persistent memory (read/write files as context grows), and heavy context engineering. So evaluating an agent means evaluating the scaffold + the model, not just the model.

Pure-reasoning & safety benchmarks

ARC-AGI tries to isolate reasoning from knowledge: tasks 100% solvable by humans but each unique, so memorization doesn't help. Pretrained LMs didn't move the needle; reasoning models (o1/o3) made it take off — ARC-AGI-1 is ~solved, -2 nearly so, -3 (interactive, 2026) still near zero. Caveats: decoupling reasoning from knowledge is hard, and it's constrained to human (not superhuman) reasoning.

ARC-AGI: infer the transformation from examples — no language, just spatial pattern reasoning. Source: ARC Prize.

Safety has no clean definition (unlike car crash-tests). HarmBench tests refusal of harmful prompts; AIR-Bench builds a risk taxonomy from regulatory frameworks. Jailbreaking (e.g. GCG's gradient-optimized prompts) bypasses refusals and even transfers from open to closed models. Safety is deeply contextual (politics, law, norms vary by country), the risks are varied (hallucination, sycophancy, abetting crime, deskilling), and many capabilities are dual-use (a cyber agent hacks or does pen-testing).

Ecological validity — does it capture real use?

Exams are far from real use; Arena prompts are real but from an uncontrolled population. Efforts to get closer at the use-case level: GDPVal (tasks from professionals across the top US-GDP occupations), MedHELM (clinical tasks from real clinicians, not standardized exams), and Clio (use LMs to analyze real user data and surface aggregate patterns, preserving privacy). The tension: realism and privacy are often at odds — the most valuable data (the real query stream) is the hardest to use.

Validity — contamination and dataset quality

"Don't train on your test set" was easy with fixed train/test splits; now models train on the whole internet and don't disclose data. Four responses to train–test overlap:

Infer overlap from the model — benchmark order should be exchangeable; if the model prefers the benchmark's specific ordering, it likely trained on it.
Reporting norms — providers should justify (like confidence intervals) that they didn't train on the test set.
Fresh evals — scrape content past the model's cutoff (LiveCodeBench, UncheatableEval); but timestamps aren't fully safe (copied code).
Private evals — internal code or personal writings never put online; especially easy for perplexity.

Also watch dataset quality: SWE-bench needed a "Verified" cleanup; benchmarks get "Platinum" audits; agentic benchmarks can be gamed (a trivial empty response scored 38% on one; incomplete test cases pass broken solutions). Tools like Docent use LMs to inspect agent traces — and you should always look at the outputs to confirm you're measuring what you think.

What's the point? And what are we evaluating?

There is no one true evaluation — the right one depends on the question: a buyer choosing model A vs B for a use case; a researcher measuring raw capability; a policymaker weighing benefits/harms; a developer getting feedback to improve. And note the shift in what is evaluated:

Era	Unit of evaluation
Pre-foundation models	methods (fixed train/test splits; only the algorithm varies) → drives algorithmic innovation
Today	models / systems (anything goes) → useful for downstream users
Exception	NanoGPT speedrun: fixed data, race the compute to a target loss → evaluates the algorithm

Either way, the imperative is the same: define the rules of the game.

Golden rules & takeaways

Evaluation = turning an abstract construct into a concrete metric — and it shapes what AI becomes, so it deserves care.
Perplexity is the natural distributional metric (great for scaling laws, "perplexity is all you need" faith), but it weights all tokens and requires trust on a leaderboard; many "accuracy" benchmarks are perplexity in disguise.
Exams saturate on a treadmill toward harder questions; they're controllable and gradeable but unrealistic and contamination-prone.
Open-ended chat needs pairwise comparisons, multiple judges, and rubrics; beware human and LLM-judge biases.
Agents = model + scaffold, graded by outcomes (unit tests, flags); the scaffold matters as much as the model.
Realism vs privacy and validity (contamination, dataset quality) are constant tensions — use fresh/private evals and audit outputs.
No one true evaluation — pick it by your goal (buy, research, policy, develop) and state the rules of the game.

Next: the data unit proper — curation, transformation, filtering, deduplication, and mixing.

References

Primary source is the lecture (executable lecture_12.py). Benchmarks named:

What does it mean for a model to be "good"?