MMMU what it measures, how it is scored and who leads it

Name: MMMU
Creator: MMMU team
Published: 2023-11-27

Multimodal · MMMU team · introduced Nov 27, 2023 · Accuracy

About this benchmark

Task: Answer college-level questions that need an image: charts, diagrams, maps, music sheets, chemical structures.
Dataset: 11,500 questions from exams, quizzes and textbooks across 30 subjects and 183 subfields.
Method: Multiple choice and short answer. MMMU-Pro removes questions a text-only model can answer and adds a vision-only setting.
Metric: Accuracy
Organization: MMMU team
Introduced: Nov 27, 2023
Paper: MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (arXiv 2311.16502)
Official leaderboard: mmmu-benchmark.github.io/
Home: github.com/MMMU-Benchmark/MMMU

Newest first. New versions are added, never rewritten.

Version	Date	What changed
MMMU-ProHarder, more robust version (arXiv 2409.02813).	Sep 4, 20242 years ago	Harder, more robust version (arXiv 2409.02813).
1Released with the paper.	Nov 27, 20232 years ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.