SWE-Bench Pro what it measures, how it is scored and who leads it

Name: SWE-Bench Pro
Creator: Scale AI
Published: 2025-09-21

Coding · Scale AI · introduced Sep 21, 2025 · Resolved

Scores from SWE-Bench Pro, published Sep 27, 2026

Rank	Model	Resolved
1	Claude Opus 5Anthropic · as “Opus 5 (Claude Code) xhigh”	98.0%
2	Claude Fable 5.1Anthropic · as “Fable 5.1 (Claude Code) high”	92.2%
3	GPT-6 AstraOpenAI · as “GPT-6 Astra (Codex) high”	90.2%
4	Claude Sonnet 5Anthropic · as “Sonnet 5 (Claude Code) xhigh”	88.2%
4	Kimi K3Moonshot AI · as “Kimi-K3 (mini-swe-agent) max”	88.2%
6	GPT-5.6 TerraOpenAI · as “GPT-5.6 Terra (Codex) xhigh”	86.3%
7	GLM-5.3Z.ai · as “GLM-5.3 (mini-swe-agent) max”	84.3%
8	GPT-5.6 SolOpenAI · as “GPT-5.6 Sol (Codex) xhigh”	82.4%
9	Gemini 3.8 FlashGoogle · as “Gemini 3.8 Flash (mini-swe-agent) high”	58.8%
10	InklingThinky · as “Inkling (mini-swe-agent) xhigh”	56.9%
11	Claude 4.5 HaikuAnthropic · as “Haiku 4.5 (Claude Code) xhigh”	25.5%

Task: Solve long-horizon, enterprise-style software engineering tasks in real repositories.
Dataset: 1,865 problems from 41 actively maintained repositories: a public set, a held-out set and a commercial set from partner startups.
Method: Agents produce a patch graded by the repository tests. V2 locks the protocol: web tools off, every diff re-graded on a pristine image.
Metric: Resolved
Organization: Scale AI
Introduced: Sep 21, 2025
Paper: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv 2509.16941)
Official leaderboard: labs.scale.com/leaderboard/swe_bench_pro_public_v2
Home: github.com/scaleapi/SWE-bench_Pro-os

Newest first. New versions are added, never rewritten.

Version	Date	What changed
V2Refreshed public split of 642 tasks across 11 repositories (89 invalid tasks dropped), co-developed with Reflection.	Sep 22, 20265 days ago	Refreshed public split of 642 tasks across 11 repositories (89 invalid tasks dropped), co-developed with Reflection.
1Released with the paper; public set from 11 repositories.	Sep 21, 20251 year ago	Released with the paper; public set from 11 repositories.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.