What happens when an artificial intelligence evaluation becomes a public game? In the case of Olmo 3, the answer was unexpected: the inputs that best guided the model toward prosocial behavior were not empathetic phrases or carefully worded instructions, but nearly unreadable strings of tokens.
The experiment, developed by Soham Padia, a master's student at Northeastern University, highlights a central problem in model evaluation: a metric can seem useful until people start optimizing it in ways its creators did not anticipate.
Steering Arena Turns Evaluation Into a Game
Padia created a test to measure how much Olmo 3 could be steered toward responses considered prosocial. In this context, the term includes qualities such as empathy, fairness, safety, privacy, and respect.
The evaluation began with 135 pairs of contrasting responses organized around 15 qualities. Each pair responded to the same message, but one response represented more prosocial behavior than the other. By comparing the model's internal activity against those examples, Padia identified a direction associated with responses more aligned with those values.
He then turned the experiment into Steering Arena, a platform where participants could submit short text prefixes. The system measured how much each input shifted Olmo 3's internal behavior toward the prosocial direction and displayed the results on a leaderboard.
Was the idea to find the kindest phrase? Not exactly. The real goal was to discover which text influenced the model most strongly, even if it made no sense to a person.
The Best Entries Looked Like Noise
In total, the experiment received nearly 600 submissions from a few dozen participants. The 36 highest-scoring entries were practically unreadable strings of tokens, with combinations such as Undert! AH :-) Rog Appl) and Angela Nombre WiBanner:] Workflow.respond-winemoji.
The best proposal written in understandable English asked Olmo 3 to respond with a short sentence full of kindness, respect, and compassion. However, it ranked 37th and received a score approximately 2.7 times lower than the winning entry.
This does not mean the strange strings were necessarily random. The system was not evaluating whether the text sounded prosocial to a human. It was measuring the change produced in certain internal signals of the model. Participants could therefore optimize directly for the behavior rewarded by the metric, even when the result was incomprehensible.
When a metric becomes public, it turns into an optimization target.
One participant took this strategy even further by using an automated method to search for higher-scoring inputs. Some successive proposals changed by only one token, while the process explored combinations that were increasingly effective for the evaluator.
What Studying Internal Signals Adds
The experiment was possible thanks to Olmo 3's level of openness. Padia worked with Olmo 3-32B through the National Deep Inference Fabric platform, known as NDIF and supported by the U.S. National Science Foundation.
NDIF made it possible to access a large model remotely without the researcher having to buy or maintain the GPUs required to run it. But remote access was not the only important factor. Olmo also provides information about its data and its later adjustment process, known as post-training.
That documentation helps address a difficult question that is hard to answer with closed models: did a particular behavior emerge during initial training, or was it added later through specialized adjustments?
Technically speaking, Padia did not have to infer everything from the model's final responses. He could observe how inputs changed the model's internal activity and publish the signal used to calculate the Steering Arena scores.
This does not make Olmo 3 a completely transparent system, but it does open a much wider window than the one available in many proprietary models. Researchers can inspect the signal, reproduce the procedure, and test whether the evaluation actually measures what it is supposed to measure.
The Lesson for People Building Models
An evaluation can indicate that a model responds safely, fairly, or empathetically under certain conditions. The problem appears when users discover how to exploit the exact criterion used to calculate that score.
Steering Arena worked as a stress test. Instead of simply checking responses carefully designed by a small team, it allowed real participants to search for unexpected ways to influence the model.
That approach can reveal flaws that a traditional evaluation would miss. It also raises a warning: an internal signal related to prosocial behavior does not automatically equal understanding, intention, or good judgment.
A model can move in the direction a metric considers positive without producing text that a person interprets as useful or respectful. The difference between those two things is precisely the kind of problem that public evaluations can bring to light.
Why Openness Matters
The case of Olmo 3 connects to a broader discussion about open models. Publishing the weights is important, but it is not always enough. Rigorous research also requires information about the data, training, later adjustments, infrastructure, and internal signals involved in the model's behavior.
With that access, a researcher can study where a particular capability or tendency might come from, rather than limiting themselves to observing inputs and outputs as if they were facing a black box.
The Steering Arena experience leaves us with a practical conclusion: if an organization wants to know how a model will behave in the real world, it is not enough to design a good test. It must also ask how users will optimize it, what shortcuts they will find, and whether the metric will continue to represent the original goal after being exposed.
AI does not always fail because it does not know the answer. Sometimes it fails because it learns to satisfy the wrong indicator. Experiments like this help detect that difference before it reaches applications with real-world consequences.