OpenAI explains why language models hallucinate | Keryc
OpenAI published a new analysis on September 5, 2025 that tries to answer a question that has probably annoyed you: why do assistants like ChatGPT sometimes invent facts with a lot of confidence? In that piece the team says it’s not just a technical bug, but largely a problem of incentives in how we train and evaluate these models. (openai.com)
What is an "hallucination" in AI and why should you care?
A hallucination is when a language model generates a plausible but false claim. Sound familiar? It can be a date, a job title, a link that doesn’t exist, or a made-up medical procedure. For everyday users and for businesses this isn’t just annoying: it can cause real mistakes and harm if it goes unnoticed. (openai.com)
"Models generate answers confidently even when they don't have the evidence to support them."
This idea is at the heart of OpenAI’s analysis: models tend to prefer giving an answer over saying "I don't know." ()
Why do they hallucinate? Two easy-to-understand causes
Evaluation incentives. If benchmarks only score the correct answer, the model learns that it’s better to gamble and guess than to abstain. In a real exam, if you don’t answer you get zero; if you answer you might get lucky. OpenAI explains that this bias toward "guessing" encourages hallucinations. (openai.com)
Training is next-word prediction. During pretraining models learn to predict next-word over piles of unlabeled text—no truth/lie tags. That works great for repetitive patterns like spelling, but for rare or long-tail facts there aren’t enough signals to learn the true answer. The best statistical strategy there is to complete with what sounds plausible, not what’s verified. (openai.com)
What about the numbers? Are they getting worse or better?
OpenAI and independent reports found something worrying: some new reasoning models (for example o3 and o4-mini) show higher hallucination rates on certain internal tests like PersonQA. TechCrunch reported that o3 hallucinated on around 33 percent of questions in PersonQA and that o4-mini reached even higher figures in some tests. External researchers also observed cases where the model claimed impossible actions, like having executed code outside its environment. (techcrunch.com, pcworld.com)
Additional reports collect differences depending on the test: in some evals the balance between abstentions and hits changes a lot depending on how you measure it. That supports OpenAI’s thesis: the way we evaluate changes model behavior. (forbes.com, openai.com)
What does OpenAI propose and what can people using AI do today?
OpenAI suggests a clear shift: reward humility and penalize confident mistakes. In practice that means updating benchmarks to give credit for abstaining when appropriate and to penalize answers that are confidently wrong. They also recommend techniques already known in the community: retrieval-augmented generation (RAG), confidence calibration, and evaluations that measure uncertainty, not just accuracy. (openai.com, arxiv.org)
For you, who use AI at work or in projects:
Ask for sources and verify: require the answer to include verifiable evidence or documents.
Use RAG: feed the model reliable documents and have the model cite those texts.
Design abstentions: craft prompts and rules so the model says "I don't know" when evidence is insufficient.
Check settings: lowering temperature, using truthfulness controls and automated tests helps, but it doesn't eliminate the problem. (arxiv.org, openai.com)
What do we learn in the end?
The main lesson is useful and also reassuring: hallucinations aren't a mystical mystery, they're the result of design choices and how we measure performance. By changing incentives, metrics and deployment practices we can reduce them a lot, though don't expect total elimination without redesigning essential parts of training and evaluation. OpenAI already proposes paths and the research community offers practical techniques you can apply today. (openai.com, arxiv.org, techcrunch.com)
If you want, I can turn this into a short list of concrete actions for your team, or into a prompt that forces the model to show sources and to abstain when it doesn't have sufficient evidence.
Stay up to date!
Get AI news, tool launches, and innovative products straight to your inbox. Everything clear and useful.
