When I read OpenAI's note about why their models went from very low scores to much better ones in ARC-AGI-3, I thought: how much does the way we use the API matter? The answer was clear and practical: it does matter — and a lot.
What happened and why it matters
ARC-AGI-3 is a 2D-games benchmark designed to measure how agents learn and reason without explicit instructions. It's a tough lab: the challenges test not only the model's raw ability, but also how you question it and how you preserve its thoughts.
OpenAI noticed that GPT-5.6 Sol, despite solving complex math problems and beating games like Pokémon FireRed in other tests, scored 7.8% on ARC-AGI-3. GPT-5.5 barely played, with 0.4%. The takeaway? It's not always that the model is bad; sometimes the issue is the evaluation harness.
The two settings that made the difference
They discovered two harness decisions that limited the model:
- The system discarded the private reasoning between actions. In other words, the model forgot its plans and internal thoughts and had to reinterpret the game every turn.
- They used a rolling-window truncation that removed older actions as the history grew, losing memory of what had already happened.
By turning on two options already used in ChatGPT and Codex, results changed radically: retaining reasoning (retained reasoning) and using compaction instead of pure truncation.
What each setting did in practice
Retaining reasoning let GPT-5.6 Sol remember its plans instead of re-evaluating everything from scratch every step. What happened? It spent less time thinking before acting and could use coherent strategies across the game.
compaction replaces the rolling window. Instead of deleting the oldest content, it summarizes and preserves relevant information as the context grows. The result: the model keeps what it learned during long matches and also produces far fewer output tokens.
Combined result: roughly 3x the score with 6x fewer output tokens.
Why this reminds us of a bigger point about evals
Public tests don't measure only the model. They measure a package: model, prompting, harness design, and API settings. A benchmark with a very generic harness can leave out capabilities that do show up in real environments because production setups preserve them.
If you recall cases where a model seems slow or erratic in a test but performs well in a demo, this might be why: the test isn't using the same deployment best practices as production.
Practical recommendations for developers
If you work with the API and want your evals to reflect the real experience better, consider:
- Use the
Responses APIinstead of older APIs. - Enable retained reasoning so private messages become useful context.
- Turn on
compactionto summarize history instead of deleting old information.
And if you compare models, look for evals that use these settings, because they better represent how models perform in real applications.
Final reflection
The answer isn't always a bigger model or more training. Sometimes it's enough to care about how we talk to it and how we preserve its thinking. That not only improves benchmark scores; it changes the experience of an agent that learns while it plays.
Original source
https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores