AI benchmarks are often presented as a definitive score: this model reasons better, that one is safer, or another follows instructions more effectively. But what happens when the same test measures several capabilities at once? BenchMIRT proposes looking beyond the final result and analyzing each question separately.
The Problem with Relying Too Much on a Single Score
A benchmark is designed with a specific goal, such as measuring reasoning, safety, or instruction following. However, each question may require additional skills that do not always appear in the evaluation’s official description.
One example is BBQ, a test created to study whether models rely on social stereotypes. One of its questions presents a grandfather and his grandchild trying to request an Uber. To answer correctly, the model must not only avoid age-related bias. It also has to identify who is who and reason from the available information instead of filling in the gaps with assumptions.
Something similar happens with WildJailbreak. The benchmark combines harmful requests, which evaluate whether a model refuses to help with damaging content, with harmless requests designed to check whether it refuses too often. Averaging both groups into a single score can hide the fact that they are measuring different capabilities.
An overall score may seem precise, but it can also hide which skill is actually driving the result.
How BenchMIRT Works
BenchMIRT is a method developed to audit benchmarks for large language models, known as LLMs. Its goal is to identify which capabilities are related to answering each question correctly and how much each task contributes to the final result.
The technique draws on Item Response Theory, or IRT, a method originally used in psychometrics. Put simply, IRT starts with an intuitive idea: not all questions have the same difficulty or provide the same information about the person answering them.
A question may be easy for nearly every model and, as a result, do little to distinguish between them. Another may clearly separate more capable models from less capable ones. BenchMIRT expands this approach through Multidimensional Item Response Theory, or MIRT, which makes it possible to analyze several capabilities at once.
The system estimates two main elements:
- Each model’s ability: how strong it is in the skills represented in the analyzed benchmarks.
- Each question’s characteristics: its difficulty and its ability to distinguish between stronger and weaker models.
An Analysis with 100 Models and More Than 34,000 Questions
To test the method, the researchers used results from 100 open-source LLMs, 16 benchmarks, and more than 34,000 questions. Six evaluations focused on general reasoning, including MMLU-Pro, GPQA, MATH, and BBH.
The other 10 were part of the Olmo 3 safety suite and included HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.
One important detail is that the researchers did not tell BenchMIRT which benchmark corresponded to each capability. The method identified two dominant dimensions on its own: safety and general reasoning.
When the analysis was repeated from scratch, the same two dimensions appeared. This suggests that the result did not depend solely on a specific configuration, but reflected a stable pattern within the dataset studied.
What It Discovered About Well-Known Evaluations
In several cases, BenchMIRT confirmed the original intention of the benchmarks. Models that performed well on reasoning tests also showed stronger general reasoning ability. Likewise, performance on jailbreak and harmful-content tests was more closely related to safety.
But some evaluations showed a more complex mix.
BBQ Measures More Reasoning Than Expected
Although BBQ is usually classified as a safety and social-bias benchmark, its results aligned much more closely with general reasoning in the BenchMIRT analysis.
This does not mean that BBQ is useless for studying bias. It means that a low score may partly reflect difficulties understanding the question, following the relationships between the characters, or reasoning from the evidence provided.
WMDP Combines Knowledge and Safety
WMDP evaluates dangerous dual-use knowledge in areas such as biology, chemistry, and cybersecurity. Some questions could help someone create a biological agent or exploit a computer system.
In this benchmark, the desired response is usually not to provide that knowledge. BenchMIRT found that its results were more closely related to general reasoning than to safety. In addition, greater reasoning ability could be associated with a lower score, because the benchmark considers it correct to refuse or withhold the dangerous information.
This detail is crucial: a low score does not always mean that the model is less capable. Sometimes it indicates that the model responded in a way the evaluation considers appropriate from a safety perspective.
HarmBench Reveals Differences Within a Single Test
HarmBench also mixed different signals. Its standard questions include requests such as writing a phishing email to steal banking information. These tasks were primarily associated with the safety dimension.
The contextual questions, which provide additional information and ask the model to act on it, showed a similar relationship with safety. By contrast, copyright-related questions, such as requesting the full lyrics of a well-known song, were more closely linked to general reasoning.
The benchmark may still be valid. The lesson is different: its total score brings together several types of behavior and should not be interpreted as a pure measure of a single capability.
Fewer Questions Could Preserve Nearly the Same Information
BenchMIRT can also identify the most informative questions in an evaluation. To do this, the researchers ranked tasks according to their ability to distinguish between strong and weak models, while maintaining a combination of easy and difficult questions.
The result was striking. Across the 16 benchmarks analyzed, keeping only 10% of the questions generally preserved an almost identical picture of which models were stronger in safety or reasoning.
When 50% of the questions were retained, the measurement matched the full evaluation more often. This opens the door to smaller, faster, and cheaper benchmarks, without necessarily losing all the useful information.
Does this mean most questions should be eliminated? Not necessarily. A reduced evaluation can be efficient, but it may also lose coverage and make it easier for a model to prepare specifically for the remaining tasks.
Predicting Responses the Model Has Not Yet Seen
BenchMIRT can use patterns learned across models and questions to estimate how a model would respond to a task that was not included in its observed results.
In the experiments, it correctly predicted whether the model would answer a held-out question correctly 79% of the time. A simpler method, which assumed that the model would perform on each question the same way it performed on the full benchmark, reached 70%.
In practice, this could reduce the need to evaluate every model on every question. If BenchMIRT already knows the model’s estimated capabilities and the demands of a task, it can make a more accurate prediction about expected performance.
Limitations and Risks
BenchMIRT is not a definitive solution. The models used to train and evaluate it had been published by March 2025, so the analysis does not show how it performs with later generations of LLMs.
In addition, the dimensions discovered depend on the benchmarks included. In this study, safety and reasoning emerged because those were the dominant capabilities in the selected set. A different group of evaluations could reveal other dimensions, such as factual knowledge, programming, or instruction following.
There is also a practical limitation. If the goal is to rank models according to their expected performance on random questions, the traditional average score may be slightly better than BenchMIRT. The new method’s advantage is that it offers a more detailed explanation of what is happening on each question.
And that transparency can have a downside. The same information that makes it possible to find the most useful questions for evaluating safety could be used to remove them and create a weaker test that is easier for an unsafe model to pass.
Understanding a benchmark better also means learning how to manipulate it better. Transparency helps, but it requires keeping evaluations robust and up to date.
Why It Matters for the Future of Evaluation
BenchMIRT puts forward a simple but powerful idea: AI tests should not be interpreted solely through their final score. An evaluation can combine reasoning, comprehension, safety, and knowledge without making that clear in the result.
By analyzing each question, researchers can detect which tasks actually measure the intended capability, which introduce additional signals, and which provide little information. This can help them design benchmarks that are more specific, efficient, and easier to interpret.
For people comparing models, the lesson is practical too. A leaderboard does not tell the whole story. Before concluding that one model is safer or smarter because of a score difference, it is worth asking what kind of questions produced that difference.
AI already has enough numbers. The next step is understanding what they mean.