Eval benchmarks — reference

Plain definition: a fixed dataset of inputs with known correct outputs, plus a scoring protocol that turns model answers into a single comparable number.

Concrete example: MMLU has 14,000 multiple-choice questions across 57 subjects. You run the model, check whether its answer matches the key, and divide correct answers by total. That fraction (e.g., 0.87) is the score.

Formal version: let $D = \{(x_i, y_i)\}_{i=1}^{N}$ be the test set with ground-truth labels $y_i$. Define a scoring function $s: (\hat{y}, y) \to \{0,1\}$. Then benchmark accuracy is $\frac{1}{N}\sum_i s(f(x_i), y_i)$, where $f$ is the model.

Why it matters: without a fixed $D$, you can't compare two models' outputs — every team would cherry-pick the examples that flatter their model.

Plain definition: a benchmark is saturated when frontier models score above ~90–95%, so the remaining gap is within noise. Two models at 96% and 97% differ by 140 examples on a 14k test set — that's one prompt-template change away from flipping the ranking.

Concrete example: MMLU launched in 2020 with humans at ~89% and GPT-3 at ~43%. By 2024, GPT-4o, Claude 3.5, and Llama 3.1 405B all exceeded 88%. The benchmark still distinguishes a random LLM from a frontier one, but it cannot distinguish any two frontier models — the numbers differ by less than measurement noise.

What saturation does not mean: a saturated benchmark isn't useless. It is a reasonable sanity-check lower bound — if a new model scores 60% on MMLU, something is badly wrong. But it should never be your headline number for a frontier model.

Plain definition: contamination occurs when benchmark examples — or very close paraphrases — appear in the model's pretraining corpus. The model may be recalling a memorized answer rather than reasoning from scratch.

Why it is so hard to avoid: modern pretraining corpora contain trillions of tokens scraped from the web. HumanEval, MATH, and GSM8K were all published publicly before most training runs. Any page that discusses them — a blog post solving an example, a GitHub repo reproducing results — potentially leaks into the corpus.

Contamination ≠ proof of cheating. A model might score high on HumanEval because it genuinely generalizes well — and also because many similar functions appeared in GitHub. You can't disentangle these effects easily. That ambiguity is exactly why contamination is so corrosive to benchmark credibility.

What it measures: Massive Multitask Language Understanding (Hendrycks 2020). 14,000 multiple-choice questions across 57 subjects: US history, anatomy, college mathematics, professional law, high-school chemistry, abstract algebra, and more. Four-option MCQ; accuracy is the metric.

Concrete example item (anatomy): "Which of the following is NOT a function of the liver? (A) Detoxification of drugs (B) Production of red blood cells (C) Synthesis of clotting factors (D) Storage of glycogen" — Answer: B.

Why it existed: In 2020, MMLU was genuinely hard — GPT-3 (175B) scored 43%. It covered an unusually broad domain slice and forced models to demonstrate knowledge across wildly different fields, not just a narrow benchmark niche.

Why it is now stale: by 2023, GPT-4 exceeded 86%. By 2024, GPT-4o, Claude 3.5 Sonnet, Llama 3.1 405B all cluster between 85–90%. The benchmark is saturated — all frontier models look the same on it.

Contamination concern: MMLU's test set is publicly downloadable. Questions appeared in web pages, Reddit threads, and study sites absorbed into pretraining corpora. High scores may partly reflect memorization of the specific 14k questions.

What changed: MMLU-Pro (2024) takes the same subject domains as MMLU but makes each question harder in two ways: (1) 10 answer choices instead of 4 — random-guessing baseline drops from 25% to 10%, (2) harder distractors chosen to be "almost right," requiring finer reasoning to eliminate.

Concrete example (college chemistry): where MMLU might ask which element is in Group 1, MMLU-Pro asks about the electronic configuration implications of a specific transition-metal complex across 10 plausible options — you cannot guess your way through.

Status: less saturated than MMLU. Top scores cluster around 75–82% for frontier models (vs 88–90% on MMLU). A model that scores 80% on MMLU-Pro is demonstrably more capable than one scoring 70%, unlike on standard MMLU.

When to cite it: Use MMLU-Pro when you need a mid-tier knowledge/reasoning check that still differentiates frontier models without requiring PhD-domain expertise in the evaluator.

What it measures: Graduate-Level Google-Proof Q&A (Rein 2023). 198 problems (the Diamond subset) in biology, chemistry, and physics, written by domain-expert PhD holders. The key design constraint: every problem was checked to be "Google-proof" — a non-expert with full internet access cannot answer it reliably in under 30 minutes. This forces models to actually reason rather than retrieve.

Concrete example (physics): A problem might describe a quantum optics experimental setup, give measured photon coincidence statistics, and ask which Bell inequality is violated and by how much — requiring Clauser–Horne–Shimony–Holt (CHSH) inequality algebra, not a fact lookup.

Why it resists contamination: Problems were written specifically for this benchmark and were never publicly posted. The Google-proof constraint means that even if a problem paraphrase exists somewhere, solving it requires multi-step domain reasoning — you can't pattern-match to a memorized derivation.

Human baselines (critical context):

Why these baselines matter: "85% on a PhD test" sounds impressive but ambiguous. Knowing that domain experts score 65% makes the milestone concrete — the model is 20 points above the best human specialists in those fields.

What it measures: HLE (Phan 2025) is a ~3000-question benchmark described as "the next MMLU" — designed to remain unsaturated for years. Problems span mathematics, natural sciences, humanities, and professional domains; they were contributed by academic experts worldwide and range from graduate-level to cutting-edge research-level difficulty.

Why it exists: The pattern is clear — every "hardest benchmark" saturates within 2–4 years of the frontier. MMLU lasted ~3 years; GPQA is already showing pressure at the top. HLE was built from the start to be much harder, with very low SOTA at launch (around 8–10% for the best available models at publication time), giving the benchmark runway to remain relevant.

Status in 2026: SOTA around 25% for the best reasoning models. The low ceiling means even a large capability jump shows as a measurable score difference — HLE will differentiate frontier models for several years.

The name: "Humanity's Last Exam" is intentionally provocative — the hypothesis is that it represents a hard ceiling on what any benchmark curated by humans can offer before AI systems exceed all human-checkable knowledge. Whether that framing holds is an open philosophical question.

What it measures: Grade School Math 8K (Cobbe 2021). 8,500 grade-school word problems requiring 2–8 arithmetic steps. Example: "Sarah has 3 apples. She buys 4 more. She gives half to her friend. How many does she have?" Every answer is a positive integer.

Why it mattered: In 2021, GPT-3 scored ~35%. Multi-step arithmetic was a genuine frontier challenge. Chain-of-thought prompting (Wei 2022) boosted GPT-3 to ~46%; instruction-tuned models jumped to ~70%+. GSM8K was the benchmark that demonstrated CoT as a technique, not a curiosity.

Why it is stale: by 2023, GPT-4 scored ~92%. By 2024, virtually every frontier model exceeds 95%, and several small models (e.g., Llama 3.1 8B with CoT) exceed 80%. The problem set has also been identified as contaminated — grade-school math problems in this style were abundant in pretraining data.

Use it as: a sanity check lower bound. If a new model scores below 70% on GSM8K, something is badly wrong.

What it measures: MATH (Hendrycks 2021). 12,500 problems from AMC 8/10/12 and AIME competitions, spanning 5 difficulty levels (1–5) and 7 subject areas: algebra, counting, geometry, number theory, pre-calculus, probability, precalculus.

Concrete difficulty example (Level 5 number theory): "Find all positive integers $n$ such that $n^2 + 4n + 4$ divides $n^4 + 6n^2 + 8$." This requires algebraic manipulation and divisibility arguments — genuine mathematical reasoning.

Progress arc: GPT-3 ~5%; GPT-4 ~52%; o1 ~94%; frontier reasoning models hit ~95%+ in 2024. The MATH benchmark is now saturated at the frontier level. Level 1–3 problems are essentially solved; only Level 5 still differentiates, and even those are near-saturated for reasoning models.

Contamination concern: MATH problems are from public competition archives. AMC/AIME problems have been discussed extensively online and in textbooks absorbed into pretraining corpora.

What it measures: AIME (American Invitational Mathematics Examination) problems from the 2024 and 2025 competitions. 15 problems per year; all answers are integers from 000–999. Problems require combinatorics, number theory, geometry, and algebra at olympiad level — substantially harder than standard MATH benchmark problems.

Concrete example (AIME 2024): "Find the largest prime $p$ such that $p$ divides $\binom{2024}{1012}$." This requires knowledge of Kummer's theorem and p-adic valuations — not derivable from keyword matching.

Why it's a natural holdout: AIME 2024 and 2025 problems were released after most current models' training cutoffs (typically late 2023). This temporal holdout makes contamination structurally impossible — the model could not have seen these specific problems during pretraining.

Current scores: o3 ~96% on AIME 2024 (the most impressive single-number result in frontier math as of 2024). Earlier models scored far lower: GPT-4 ~30–40%, o1 ~74%. The jump from o1 to o3 on AIME is one of the clearest illustrations of test-time compute scaling.

AIME 2025: harder than 2024; o3 scores lower, providing continued differentiation even at the frontier.

What it measures: FrontierMath (Glazer 2024, Epoch AI). A curated set of novel, research-level mathematics problems contributed by professional mathematicians. Problems are computationally verifiable (final answer is a number or formal object) but require graduate or research-level insight to solve. Problems are held private — never released publicly.

Concrete example type: "Compute the number of irreducible degree-7 polynomials over $\mathbb{F}_{13}$ whose Galois group over $\mathbb{F}_{13}$ is isomorphic to the cyclic group $C_7$." This requires algebraic number theory knowledge that no reasonable web page summarizes in a solve-able form.

The headline result: Before o3, SOTA on FrontierMath was below 3% — even the best models could barely solve 1 in 33 problems. o3 jumped to approximately 25%. This gap — 3% → 25% — was the most cited 2024 frontier math result. It demonstrates that test-time compute with strong reasoning can unlock research-level mathematical capability.

Anti-contamination mechanism: Problems are held private by Epoch AI; they were never posted on any public platform. Unlike AIME, FrontierMath problems don't appear on the internet at all, not even after the evaluation — a continuous supply of new private problems is maintained.