SWE-Lancer what it measures, how it is scored and who leads it

Name: SWE-Lancer
Creator: OpenAI
Published: 2025-02-17

Coding · OpenAI · introduced Feb 17, 2025 · Dollars earned

About this benchmark

Task: Complete real freelance software jobs from Upwork, both implementation and choosing between proposals.
Dataset: Over 1,400 tasks worth $1 million in real payouts, from $50 bug fixes to $32,000 features.
Method: Implementation tasks graded by end-to-end tests written by engineers; managerial tasks against the original manager's choice.
Metric: Dollars earned
Organization: OpenAI
Introduced: Feb 17, 2025
Paper: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (arXiv 2502.12115)
Home: github.com/openai/SWELancer-Benchmark

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Feb 17, 20251 year ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.