BIG-Bench Hard what it measures, how it is scored and who leads it

Name: BIG-Bench Hard
Creator: Google
Published: 2022-10-17

Reasoning · Google · introduced Oct 17, 2022 · Accuracy · saturated

About this benchmark

Task: Solve 23 BIG-Bench tasks where earlier models fell short of average human raters.
Dataset: 23 tasks from BIG-Bench: logical deduction, date understanding, multistep arithmetic and more.
Method: Three-shot prompts with and without chain of thought; exact match.
Metric: Accuracy
Organization: Google
Introduced: Oct 17, 2022
Paper: Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (arXiv 2210.09261)
Home: github.com/suzgunmirac/BIG-Bench-Hard

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Oct 17, 20223 years ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.