BigCodeBench what it measures, how it is scored and who leads it

Name: BigCodeBench
Creator: BigCode
Published: 2024-06-22

Coding · BigCode · introduced Jun 22, 2024 · pass@1

About this benchmark

Task: Write Python that calls many library functions to complete practical tasks.
Dataset: 1,140 tasks using 139 libraries across 7 domains, about 5.6 tests each.
Method: Code is run against the tests (branch coverage about 99%).
Metric: pass@1
Organization: BigCode
Introduced: Jun 22, 2024
Paper: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (arXiv 2406.15877)
Official leaderboard: bigcode-bench.github.io/
Home: github.com/bigcode-project/bigcodebench

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Jun 22, 20242 years ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.