AI benchmarks are often reduced to a single figure: one model scores 80%, another 75%, and the first one appears to be better. But what is that percentage really measuring? A new methodology from Ai2, called BenchMIRT, analyzes every question in an evaluation to separate the capabilities that influence the result.
A benchmark can measure more than one skill
A benchmark is designed with a specific goal: evaluating reasoning, safety, instruction following, or specialized knowledge. The problem is that its questions may require additional capabilities that do not always appear in the name of the test.
For example, BBQ is designed to analyze whether a model relies on social stereotypes. One of its questions presents a grandfather and his grandchild trying to order a vehicle. To answer correctly, the model must not only avoid age-related bias. It also has to identify who is who, follow the information provided, and reason without filling in gaps with assumptions.
This means that a low score could reflect a safety problem, but it could also point to difficulties with comprehension or reasoning. If everything is blended into a single average, interpreting the result becomes more difficult.
Something similar happens with WildJailbreak. The benchmark combines harmful requests, which evaluate whether the model rejects dangerous content, with benign requests designed to check whether it rejects too much. The first group is more closely related to safety, while the second may depend more on general reasoning.
A single score can hide the fact that different questions are measuring different capabilities.
How BenchMIRT works
BenchMIRT takes Item Response Theory, known as IRT, as its starting point. This technique was originally used in psychometrics. In simple terms, the theory recognizes that not all questions provide the same information.
Some questions are more difficult. Others are better at distinguishing between strong and weak participants. BenchMIRT expands on this idea through Multidimensional Item Response Theory, or MIRT, to detect multiple capabilities that may influence a response.
The method analyzes two levels:
- The model: estimates its strengths in the capabilities present in the selected benchmarks.
- The question: calculates its difficulty and how well it distinguishes between models with greater or lesser mastery of those capabilities.
To test it, Ai2 trained BenchMIRT using results from 100 open-source language models, 16 benchmarks, and more than 34,000 questions. Six evaluations focused on general reasoning, including MMLU-Pro, GPQA, MATH, and BBH. The other ten belonged to the Olmo 3 safety suite and included HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.
What is interesting is that the researchers did not tell the system which benchmark corresponded to each capability. Even so, BenchMIRT identified two dominant dimensions: safety and general reasoning. When the analysis was repeated from scratch, the same dimensions appeared again, suggesting that the result was stable.
What it discovered about current benchmarks
In many cases, BenchMIRT confirmed the original purpose of the evaluations. Reasoning benchmarks were mainly associated with that capability, while jailbreak and harmful-request tests were linked to safety.
But some less intuitive results also appeared.
BBQ seems to measure more reasoning than safety
Although BBQ is usually classified as a safety and social-bias benchmark, its score showed a much stronger correlation with general reasoning. BenchMIRT found a correlation of 0.85 with reasoning and just 0.06 with safety.
In practice, a low BBQ result could say more about a model’s ability to understand and solve the questions than about its tendency to reproduce stereotypes.
WMDP works in the opposite direction
WMDP evaluates dangerous dual-use knowledge in areas such as biology, chemistry, and cybersecurity. In this case, the desired response is usually not to provide the risky information.
BenchMIRT detected a correlation of -0.89 between the WMDP score and general reasoning, with no significant relationship to safety. The reason is important: models with greater knowledge and reasoning ability may be more capable of answering dangerous questions, which lowers their score if they do not reject those requests.
A high WMDP score does not necessarily mean that the model reasons better. It may indicate that the model lacks certain kinds of dual-use knowledge or avoids providing the requested information.
HarmBench mixes different signals
HarmBench evaluates whether a model complies with harmful requests. Its standard questions, such as writing a phishing email, aligned mainly with safety. The same was true of contextual questions, which provide additional information and ask the model to act on it.
However, copyright-related questions, such as requesting the full lyrics of a well-known song, showed a much weaker relationship with safety. Within a single benchmark, then, there are groups of questions that may be examining different behaviors.
Other notable results
- MMLU-Pro and BBH were strongly associated with general reasoning, with correlations of 0.97 and 0.94, respectively.
- WildJailbreak, JailbreakTrigger, and Do-Anything-Now were mainly related to safety, with correlations close to -0.90.
- XSTest was split between reasoning and safety, which makes sense given its combination of questions about over-refusal and safety.
- ToxiGen showed a weak relationship with both capabilities. Ai2 notes that it may be saturated, since the models reached an average score of 92%.
These data do not prove that the benchmarks are poorly designed. They reveal that their scores may combine several signals and that they need to be interpreted more carefully.
Fewer questions without losing too much information
BenchMIRT can also identify which questions are most useful for distinguishing between strong and weak models in a specific capability.
The researchers ranked the questions from the 16 benchmarks according to the information they provided and kept a combination of easy and difficult items. When using only 10% of the questions, the method generally preserved a very similar picture of which models were stronger in safety or reasoning.
With 50% of the questions, the measurement often matched the full benchmark even more closely. This could help create smaller, faster, and less expensive evaluations, especially when testing each model with thousands of questions becomes costly.
BenchMIRT also attempted to predict whether a model would answer a question correctly when it had not seen that question during the analysis. It was correct in 79% of cases. A simpler method, which assumed that the model would perform on every question at the same level as its overall average, reached 70%.
The difference does not turn BenchMIRT into a crystal ball. But it does show that knowing a model’s abilities and the demands of each question can produce more accurate estimates.
Limitations and the risk of optimizing too much
The methodology has limitations. The models used in the study had been published through March 2025, so the analysis does not yet show how it performs with more recent generations of language models.
In addition, the dimensions discovered depend on the benchmarks included. In this experiment, safety and reasoning appeared because those were the dominant capabilities in the selected set. With other evaluations, different dimensions could emerge, such as factual knowledge, programming, or multimodal understanding.
There is also an important trade-off. If the goal is to rank models according to their performance on randomly selected questions, the traditional average score may be slightly better than BenchMIRT. The advantage of the new methodology is that it offers a more detailed explanation of what happens at the level of each question.
And that transparency can be used for better or worse. The same estimates that make it possible to find the most informative safety questions could be used to remove those questions and build an evaluation that is easier for an unsafe model to pass.
Measuring better is not just about obtaining a more precise number. It also means understanding what that number represents and what it might be hiding.
BenchMIRT points toward a more specific and transparent way to evaluate AI models. Instead of treating each benchmark as a black box, it analyzes which questions drive the result, which capabilities are involved, and which questions provide little information.
For people developing models, this could mean more efficient tests. For researchers and users, it offers something even more important: a way to ask not only which model achieved the highest score, but what skill it is actually demonstrating.