MRCR (multi-round coreference) what it measures, how it is scored and who leads it

Name: MRCR (multi-round coreference)
Creator: Google DeepMind; open version by OpenAI
Published: 2024-09-19

Long context · Google DeepMind; open version by OpenAI · introduced Sep 19, 2024 · Match ratio

About this benchmark

Task: Find the right one of several near-identical requests hidden in a long conversation and reproduce its answer.
Dataset: Synthetic long conversations with repeated similar requests.
Method: Similarity between the reproduced and reference answer, reported by context length.
Metric: Match ratio
Organization: Google DeepMind; open version by OpenAI
Introduced: Sep 19, 2024
Paper: Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries (arXiv 2409.12640)
Home: huggingface.co/datasets/openai/mrcr

Newest first. New versions are added, never rewritten.

Version	Date	What changed
Michelangelo MRCRIntroduced in the Michelangelo paper.	Sep 19, 20242 years ago	Introduced in the Michelangelo paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.