AI benchmarks: what each one measures

4 of 85 benchmarks

Benchmark	Domain	Introduced	Paper	Scores
RULERRetrieve, trace and aggregate information across long synthetic contexts, beyond one needle.Long context · 2024NVIDIA	Long context	Apr 9, 20242 years ago	2404.06654	·
LongBenchUnderstand long real documents in English and Chinese: QA, summarisation, code, few-shot.Long context · 2023Tsinghua University and Zhipu AI	Long context	Aug 28, 20233 years ago	2308.14508	·
∞Bench (InfiniteBench)Answer questions over contexts longer than 100,000 tokens: books, code, math, dialogue.Long context · 2024Tsinghua University and collaborators	Long context	Feb 21, 20242 years ago	2402.13718	·
MRCR (multi-round coreference)Find the right one of several near-identical requests hidden in a long conversation and reproduce its answer.Long context · 2024Google DeepMind; open version by OpenAI	Long context	Sep 19, 20242 years ago	2409.12640	·

Scores counts the models with a captured score on the boards that publish this benchmark. A dot means we capture no board for it yet.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.

AI benchmarks: what is being measured