MMLU-Pro what it measures, how it is scored and who leads it

Name: MMLU-Pro
Creator: TIGER-Lab (University of Waterloo)
Published: 2024-06-03

Knowledge · TIGER-Lab (University of Waterloo) · introduced Jun 3, 2024 · Accuracy

Scores from MMLU-Pro, published Mar 11, 2026

Rank	Model	Accuracy
1	Gemini 3.1 Pro PreviewGoogle · as “Gemini-3.1-Pro”	91.2%
2	Gemini 3 ProGoogle · as “Gemini-3-Pro(11/25)”	90.1%
3	o1OpenAI · as “GPT-o1”	89.3%
4	Claude Opus 4.6Anthropic · as “Claude-4.6-Opus(Thinking)”	89.1%
5	Gemini 3 FlashGoogle · as “Gemini-3-Flash(12/25)”	88.6%
6	MiniMax M2.1MiniMax · as “MiniMax-M2.1”	88.0%
7	Qwen3.5 397B A17BAlibaba · as “Qwen3.5-397B-A17B”	87.8%
8	Seed2.0 LiteByteDance · as “Seed2.0-Lite”	87.7%
9	GPT-5.4OpenAI · as “GPT-5.4”	87.5%
10	Claude Sonnet 4.5Anthropic · as “Claude-4.5-Sonnet(Thinking)”	87.4%
10	GPT-5.2OpenAI · as “GPT-5.2”	87.4%
12	Claude Opus 4Anthropic · as “Claude-4-Opus-Thinking”	87.3%
12	Claude Opus 4.5Anthropic · as “Claude-4.5-Opus(Thinking)”	87.3%
12	Claude Sonnet 4.6Anthropic · as “Claude-4.6-Sonnet(Thinking)”	87.3%
15	Hunyuan T1Tencent · as “Hunyuan-T1”	87.2%
16	GPT-5OpenAI · as “GPT-5(high)”	87.1%
16	K2.5 1t A32B · as “K2.5-1T-A32B”	87.1%
18	Grok 4SpaceXAI · as “Grok-4”	87.0%
18	Seed V1.5ByteDance · as “Seed-Thinking-v1.5”	87.0%
18	Seed2.0 ProByteDance · as “Seed2.0-Pro”	87.0%
21	Qwen3.5 122B A10BAlibaba · as “Qwen3.5-122B-A10B”	86.7%
22	Seed1.6ByteDance · as “Seed1.6-Thinking”	86.6%
22	Seed1.6 BaseByteDance · as “Seed1.6-Base”	86.6%
24	GPT-5.1OpenAI · as “GPT-5.1”	86.4%
24	Seed1.6 AdaByteDance · as “Seed1.6-Ada-Thinking”	86.4%

Task: Answer ten-option multiple-choice questions that need more reasoning than MMLU.
Dataset: About 12,000 questions across 14 subjects, MMLU questions filtered and extended with harder ones.
Method: Chain-of-thought prompting, five-shot; the chosen option is compared with the key. Ten options make guessing worth 10%.
Metric: Accuracy
Organization: TIGER-Lab (University of Waterloo)
Introduced: Jun 3, 2024
Paper: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (arXiv 2406.01574)
Official leaderboard: huggingface.co/spaces/TIGER-Lab/MMLU-Pro
Home: github.com/TIGER-AI-Lab/MMLU-Pro

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Jun 3, 20242 years ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.