OSWorld what it measures, how it is scored and who leads it
Agents and tools · University of Hong Kong (XLang) · introduced Apr 11, 2024 · Success rate
About this benchmark
- Task
- Operate a real computer (Ubuntu, Windows, macOS) through screenshots and input to finish open-ended tasks.
- Dataset
- 369 tasks with real web and desktop apps, file I/O and multi-app workflows.
- Method
- Execution-based checking scripts per task. People completed 72%, the best model 12% at release.
- Metric
- Success rate
- Organization
- University of Hong Kong (XLang)
- Introduced
- Apr 11, 2024
- Official leaderboard
- os-world.github.io/
Versions
Newest first. New versions are added, never rewritten.
| Version | Date | |
|---|---|---|
| 1Released with the paper. | Apr 11, 20242 years ago | Released with the paper. |
Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.