Glossary · Category
Evaluation Benchmarks
25 plain-English definitions.
- AlpacaEval
- An automated benchmark that uses an LLM judge to compare model responses against a reference model's responses, producing a win-rate leaderboard.
- ARC (AI2 Reasoning Challenge)
- A benchmark of grade-school science questions designed to require genuine reasoning rather than simple retrieval.
- Arena-Hard
- A benchmark derived from Chatbot Arena prompts, curated for difficulty and used with LLM-as-judge scoring to rank chat model quality.
- BFCL (Berkeley Function-Calling Leaderboard)
- A benchmark measuring how accurately models generate correct structured function/tool calls given an API specification.
- BIG-bench
- A large, crowd-sourced collection of diverse tasks contributed by researchers to probe emergent LLM capabilities across many domains.
- Chatbot Arena / LMArena
- A crowdsourced platform where users compare anonymous model outputs head-to-head, producing an Elo-style leaderboard of human preference.
- Contamination (benchmark/data)
- When a model's training data overlaps with benchmark test questions, inflating reported scores in a way that doesn't reflect true generalization.
- DROP
- A reading-comprehension benchmark requiring discrete numerical reasoning over paragraphs (e.g., counting, sorting, arithmetic on extracted values).
- Elo rating
- A relative skill-ranking system (borrowed from chess) used by arenas like Chatbot Arena to rank models based on pairwise human preference wins.
- GPQA (Graduate-Level Google-Proof Q&A)
- A benchmark of very hard, expert-written science questions designed to be resistant to simple web lookup, used to test frontier-level reasoning.
- GSM8K
- A benchmark of grade-school-level math word problems used to measure basic multi-step arithmetic reasoning.
- HellaSwag
- A commonsense-reasoning benchmark requiring models to pick the most plausible continuation of an everyday scenario.
- IFEval (Instruction-Following Eval)
- A benchmark that checks whether models satisfy precise, verifiable formatting/content instructions (e.g., "answer in exactly 3 bullet points").
- LiveCodeBench
- A contamination-resistant coding benchmark that continuously adds problems from recent programming contests to avoid models having memorized answers.
- MATH
- A benchmark of competition-level mathematics problems spanning algebra, geometry, and calculus, harder than GSM8K.
- MBPP (Mostly Basic Python Problems)
- A benchmark of entry-level Python programming tasks used to evaluate code generation, similar in spirit to HumanEval but broader in coverage.
- MMLU (Massive Multitask Language Understanding)
- A benchmark of multiple-choice questions spanning 57 academic subjects, widely used to measure broad knowledge and reasoning.
- MMLU-Pro
- A harder, more robust revision of MMLU with more answer choices and reasoning-focused questions, designed to reduce score saturation among top models.
- MMMU (Massive Multi-discipline Multimodal Understanding)
- A benchmark of college-level multimodal (text+image) questions across many academic disciplines, used to evaluate vision-language models.
- MT-Bench
- A multi-turn conversational benchmark that uses an LLM judge to score model responses across varied instruction-following categories.
- SWE-bench Verified
- A human-filtered subset of SWE-bench with verified solvable issues, used as a more reliable agentic coding benchmark than the full set.
- Tau-bench
- A benchmark that evaluates agentic tool-use models on realistic multi-turn customer-service-style tasks requiring policy adherence and tool coordination.
- TruthfulQA
- A benchmark measuring whether models avoid generating common human misconceptions and false claims, testing truthfulness rather than pure knowledge.
- WebArena
- A realistic web-browsing benchmark that tests whether agents can complete multi-step tasks on simulated live websites.
- WinoGrande
- A large-scale commonsense-reasoning benchmark based on pronoun-resolution puzzles designed to reduce spurious statistical shortcuts.