QAH improves a 4-bit AI model over BF16

A language model compressed by half and reduced to just 4 bits managed to outperform its own bfloat16 version on 7 of 9 benchmarks. The result comes from Quantization-Aware Healing, a recovery technique developed by Multiverse Computing to reduce the cost of running large models without automatically accepting a loss in quality.

Can a smaller model with less precision be better than the model it came from? The usual answer would be no. However, the results presented by the team show that compression does not have to be only a sacrifice: it can also become a new stage of learning.

What problem does QAH try to solve

Large models usually go through several stages before reaching production:

1. Their architecture is reduced, for example by removing layers, attention heads, or neurons.
2. Their weights are quantized, meaning they are represented with fewer bits to save memory and compute.
3. An attempt is made to recover some of the lost capabilities through additional training.

Traditional quantization can turn a 16-bit model into an 8- or 4-bit model, but those savings usually come with some degradation. When the architecture has also been reduced, the problem becomes more complex: it is no longer just about representing the same weights with less precision, but about transferring capabilities to a network with fewer parameters.

The most common technique for repairing this damage is quantization-aware training, or QAT. This method simulates quantization during training and adjusts the weights using a task-based loss. In simple terms, the model tries to relearn how to respond correctly while operating with the compressed representation.

The drawback is the cost. QAT may require repeating expensive adjustment stages, such as supervised training, instruction optimization, or agent adaptation. In addition, the model can become unstable if training continues after it has reached its best performance.

The difference: teaching from the original model

Another option is quantization-aware distillation, known as quantization-aware distillation, or QAD. Instead of learning only from correct answers, the compressed model tries to imitate the probability distribution produced by a complete, frozen model.

This works well when quantization is the only change. But if parameters are removed first, the complete reference model no longer has exactly the same architecture as the student. In that scenario, using the reduced model in bfloat16 as the teacher means learning from a version that has already lost part of the original information.

QAH changes that starting point. The technique distills directly from the original model, before structural compression, into the smaller, quantized model. The teacher may have 120 billion parameters and operate at high precision, while the student has 60 billion and uses the 4-bit MXFP4 format.

The teacher and student do not need to share an architecture to transfer knowledge. It is enough to compare the output distributions they produce for the same data.

To do this, QAH uses a loss based on KL divergence applied to the logits, the values the model generates before converting them into probabilities. The student does not receive only a hard label, such as "the correct answer is B," but a richer distribution indicating how much the original model considered each alternative.

Why this strategy can be more stable

Distillation toward a fixed teacher has a natural limit: when the student approaches the teacher's distribution, the pressure to keep changing decreases. In contrast, a loss based on labels can continue pushing the weights for too long and end up damaging capabilities the model had already learned.

This difference matters in production. A model trained with QAT may require very careful monitoring to stop at exactly the right moment. According to Multiverse Computing's experiments, QAH reaches its best result sooner and preserves its performance for longer.

A 4-bit model outperforms its BF16 version

The team applied QAH to a GPT-OSS model with 120 billion parameters. After compressing it to 60 billion, they recovered it in bfloat16 and finally quantized it to MXFP4 using QAH.

The main comparison was made between the 60-billion-parameter QAH model and its 60-billion-parameter bfloat16 equivalent, considered the best full-precision version available for that reduced architecture.

The 4-bit model outperforms its bfloat16 version on 7 of the 9 tests. Its largest gains appear precisely in areas that usually suffer most from compression: reasoning over long contexts and mathematics.

On AA-LCR, the QAH model improves by 7.4 points. On AIME 2025, it advances by 5.6 points. It falls behind only on MMLU-Pro and SciCode, with differences of less than 1.5 points.

The comparison with the original 120-billion-parameter teacher is also striking. The QAH model reaches 66.5 versus the teacher's 66.0 on LiveCodeBench. On GPQA Diamond, it remains relatively close, with 67.4 versus 69.0, although it retains a larger gap in extreme-context reasoning.

QAH reaches its best performance faster than QAT

To separate the effect of the loss function from the rest of the process, the team compared QAH and QAT on a 9-billion-parameter GPT-OSS model quantized to MXFP4.

Both methods reached similar peaks on a combination of MMLU-Pro, LiveCodeBench, and GPQA Diamond: 54.9 points for QAH and 54.6 for QAT. The difference appeared in speed and subsequent stability.

• QAH reached its maximum in approximately 100 steps.
• QAT needed nearly 700 steps to reach its best result.
• QAH stayed close to its maximum for the remainder of training.
• QAT lost almost 19 points by step 1,200.

This does not mean that QAH is automatically superior for every model or dataset. It does indicate that the choice of loss function can significantly change training ease and the risk of deploying a checkpoint that has already begun to degrade.

Less memory and lower inference cost

The benefit is not limited to benchmarks. Operating at 4 bits, the QAH model uses approximately four times less memory for its weights than the equivalent bfloat16 version.

In addition, because it has half as many parameters as the 120-billion-parameter teacher, it requires approximately half the compute per token. In model families that are normally distributed in bfloat16, combining parameter reduction with quantization could approach an eightfold reduction in compute per token.

The practical consequence is clear: models that previously required expensive hardware could run on smaller infrastructures. This could reduce the cost of serving assistants, programming tools, and systems that use long contexts.

The challenge of 32,000-token contexts

QAH must also work with lengthy documents. To prevent training from consuming too much memory, the team reused a blockwise KL-divergence loss developed in earlier work on efficient distillation.

Instead of building a huge matrix containing all sequence positions and the entire vocabulary at once, the calculation processes successive chunks. This allows training to handle sequences of up to 32,000 tokens within a fixed GPU memory budget.

This implementation detail may seem minor, but it is essential for real-world applications. A model that only works well with short texts has limited usefulness for document analysis, programming, and agents that must retain information across many interactions.

What this means for the future of compressed models

QAH's results suggest that quantization does not have to be merely a later stage that introduces losses. If a new distillation is performed from the original model, the process can transfer information that the first recovery failed to preserve.

Even so, the data should be interpreted carefully. The results apply to specific models, methods, and benchmarks, and it will still be necessary to test how QAH behaves with other architectures, domains, sizes, and datasets. It will also be important to measure the full cost of precomputing the teacher's logits and running the recovery process at scale.

The central idea is powerful: a smaller model does not have to be an impoverished copy of a large one. With the right distillation strategy, it can recover capabilities, reduce costs, and, in some cases, outperform the full-precision version of its own architecture.

Original source

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing