MuSR what it measures, how it is scored and who leads it

Name: MuSR
Creator: University of Texas at Austin
Published: 2023-10-24

Reasoning · University of Texas at Austin · introduced Oct 24, 2023 · Accuracy

About this benchmark

Task: Solve long natural-language murder mysteries, object placements and team allocations step by step.
Dataset: Algorithmically generated narratives with a hidden reasoning tree behind each question.
Method: Multiple choice after reading the story; chain of thought allowed.
Metric: Accuracy
Organization: University of Texas at Austin
Introduced: Oct 24, 2023
Paper: MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (arXiv 2310.16049)
Home: github.com/Zayne-sprague/MuSR

Newest first. New versions are added, never rewritten.

Version	Date	What changed
1Released with the paper.	Oct 24, 20232 years ago	Released with the paper.

Sources: each benchmark's paper (arXiv) and its official site or leaderboard. Scores are the boards captured here every day, exactly as published.