MultiChallenge what it measures, how it is scored and who leads it

Name: MultiChallenge
Creator: Scale AI
Published: 2025-01-29

Chat and instructions · Scale AI · introduced Jan 29, 2025 · Average accuracy

Leaderboard Paper

Scores from Scale SEAL MultiChallenge, published Nov 11, 2026

Rank	Model	Score
1	GPT-6.1 SolOpenAI · as “gpt 6.1 Sol”	82.0%
2	Gemini 4 ArgonGoogle · as “Gemini 4 Argon”	78.7%
3	Muse SparkMeta · as “Muse Spark”	75.5%
4	Muse Spark 1.1Meta · as “Muse Spark 1.1”	75.3%
5	DeepSeek V4P1 FlashDeepSeek · as “Deepseek V4p1 Flash 1017”	74.9%
6	GPT-6.1 TerraOpenAI · as “GPT 6.1 Terra”	74.2%
6	Muse Spark OpenMeta · as “Muse Spark Open”	74.2%
8	Gemini 3.1 Pro PreviewGoogle · as “gemini-3.1-pro-preview”	71.4%
9	GPT-5.4 ProOpenAI · as “gpt-5.4-pro-2026-03-05”	69.2%
10	Claude Haiku 5.5Anthropic · as “Claude Haiku 5.5”	67.7%
11	Gemini 3 ProGoogle · as “gemini-3-pro-preview”	65.7%
12	GPT-5.1OpenAI · as “gpt-5.1-2025-11-13-thinking”	63.4%
13	GPT-5OpenAI · as “gpt-5-thinking”	63.2%
14	o3-proOpenAI · as “o3-pro-2025-06-10-reasoning-high”	62.4%
15	Kimi K2.5Moonshot AI · as “kimi-k2.5”	61.4%
16	Gemini 3.1 Flash-LiteGoogle · as “gemini-3.1-flash-lite-preview”	60.6%
17	GPT-5 miniOpenAI · as “gpt-5-mini-thinking”	59.0%
18	Claude Opus 4.5Anthropic · as “claude-opus-4-5-20251101-thinking”	59.0%
19	Claude Opus 4Anthropic · as “claude-4-opus-thinking”	58.6%
20	Claude Opus 4.1Anthropic · as “claude-opus-4-1-20250805-thinking”	57.2%
21	Claude Sonnet 4Anthropic · as “claude-4-sonnet-thinking”	57.1%
22	o3OpenAI · as “o3-2025-04-16-reasoning-high”	56.6%
23	Claude Opus 4.6Anthropic · as “claude-opus-4-6 (Non-Thinking)”	56.0%
24	Kimi K2 ThinkingMoonshot AI · as “kimi-k2-thinking”	55.4%
25	Claude Sonnet 4.5Anthropic · as “claude-sonnet-4-5-20250929-thinking”	55.3%

About this benchmark

Task: Hold realistic multi-turn conversations: retain instructions, remember context, stay self-coherent.
Dataset: Human-written multi-turn conversations in four challenge categories.
Method: Graded with instance-level rubrics by a model judge checked against people.
Metric: Average accuracy
Organization: Scale AI
Introduced: Jan 29, 2025
Paper: MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs (arXiv 2501.17399)
Official leaderboard: labs.scale.com/leaderboard/multichallenge

Versions

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Jan 29, 20251 year ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.