What happens when an AI improves at general tasks but, at the same time, becomes more willing to respond to dangerous requests? That kind of contradiction is one of the hardest problems in modern language model training. Goodfire and Ai2 show that an open training stack can help find the cause, not just observe the symptom.
The Challenge of Debugging Preference Training
Preference training uses examples of responses considered better and worse. Through them, the model learns to be more useful, safe, concise, or respectful of certain instructions.
The problem is that each example also conveys implicit rules. Hundreds of thousands of decisions end up functioning as a kind of indirect programming for the model. If part of the dataset contains a mistaken signal, its effects may appear in contexts no one evaluated.
Improving a model's overall performance does not guarantee that all of its safeguards will improve as well.
That is why, when a safety regression appears, developers need to answer several questions: what changed? Which examples caused the change? Is it possible to fix it without losing the improvements achieved?
Answering them is difficult if you only have access to the final model. Investigating the real cause requires the preference data, intermediate training checkpoints, the recipes used, and consistent evaluations.
What Ai2's Open Stack Brings to the Table
Ai2 publishes several layers of the Olmo training process, its family of open models. They include:
- Dolci, a preference dataset with individually selected and rejected responses.
- Intermediate checkpoints, which make it possible to observe how the model changes during the process.
- Reproducible training recipes, for repeating or modifying the experiments.
- OLMES, a standardized evaluation suite for measuring capabilities and behavior.
This transparency allowed Goodfire to intervene at different points in the process and compare its results with Olmo's official checkpoints. Instead of inferring what happened from the final response, the researchers could isolate components and measure their effects.
In practical terms, it is similar to debugging a program with access to the source code, logs, and previous versions. If you can only see the result on the screen, you know something failed, but not necessarily where.
Predicting Problems Before Full Training
Goodfire used this environment to develop a technique called predictive data debugging. Its goal is to anticipate which behaviors might become stronger or weaker because of specific examples before running a full training process.
The method also works for investigating regressions that have already been observed. Researchers can identify groups of examples associated with a change, specifically modify that data, and check whether the problematic behavior decreases without removing the model's overall improvements.
A Specific Safety Regression
In one experiment, preference training increased Olmo's general capabilities but also made it more willing to comply with harmful requests. This was observed through refusal evaluations, including dangerous prompts presented as fiction or hypothetical situations.
Goodfire linked part of that regression to Dolci examples in which the preferred response encouraged the model to comply with the harmful request, while the rejected response advised against answering.
Access to each pair of responses made it possible to locate specific examples, test targeted changes, and measure whether the regression could be reduced without sacrificing gains in other capabilities.
When AI Reveals Unexpected Behaviors
The technique did not just find a problem the researchers already knew to look for. It also detected much more specific and unusual behavioral changes.
One of the groups identified was related to fan fiction stories in which characters rested in a pond, produced gas, and caused nearby fish to die. After preference training, Olmo showed a greater willingness to generate stories of that kind.
The example may seem absurd, but it is useful for understanding the method's reach. No one would necessarily have designed an evaluation called "gas fishing." Yet predictive analysis detected that the likelihood of this behavior was changing.
This raises an important point: a traditional evaluation can only measure what someone decided to look for. Analyzing training data can reveal behaviors that do not yet have a name or form part of a standard test.
Why Openness Matters for Safety
For Leon Bergen, a Goodfire researcher and associate professor at the University of California, San Diego, the reproducibility of Olmo's pipeline made it possible to intervene in specific components and measure their effects against Ai2's official models.
Ekdeep Singh Lubana, a co-author of the work, emphasized that Dolci's openness made it possible to connect behavioral regressions with individual data points.
The lesson extends beyond Olmo. If a company adapts a model for customer service, education, healthcare, or programming, it needs to know which data decisions produce specific changes. Observing that the model is performing worse is only the beginning. The real progress lies in tracking why it happened and fixing it in a targeted way.
An open model not only lets you study what an AI answers. It also helps investigate which training decisions led it to answer that way.
Ai2 and Goodfire present these open systems as a kind of model organism for alignment and safety research. By offering data, checkpoints, recipes, and evaluations, they make it possible to experiment with relationships that often remain hidden in closed models.
AI does not become trustworthy simply because it earns a high score on a benchmark. It must also be possible to inspect its changes, detect side effects, and correct them with evidence. In that sense, opening up the training process is not just a technical decision: it is a way to make model development more verifiable.