What happens when less than 3% of requests consume almost half of the processing time? We are not talking about a crashed server or an especially complex query. We are talking about language models that enter a repetition loop and continue generating text until they reach the maximum token limit.
This phenomenon, known as text degeneration, is usually treated as a quality problem. However, an analysis applied to OCR systems shows that it is also a problem of performance, cost, and operational capacity.
When a response gets stuck
An autoregressive model generates text token by token. Under normal conditions, it produces the response and emits a special token called EOS, which indicates that the sequence has ended.
A degenerate request never reaches that point correctly. The model begins repeating a token, a phrase, or an entire fragment. The sequence continues until the server activates the max_tokens limit and forcibly stops the generation.
The problem is not just that the result is useless. A response that should finish in a few seconds can keep the GPU busy for much longer than a healthy request with a similar input.
The defective request does not pay the entire cost on its own. The rest of the queue ends up paying it too.
A small failure that slows down the entire system
The team behind DharmaOCR observed this behavior while evaluating a model specialized in OCR for PDF documents. In one experiment, fewer than 3% of the pages generated degenerate sequences, but those requests consumed nearly half of the total processing time.
When the degenerate requests were replaced with others of average duration, total inference time fell from 7.3 minutes to 4.2 minutes. According to the analysis, the loops inflated the batch's execution time by 42.47%.
The explanation lies in how modern inference servers work. Tools such as vLLM group multiple requests into dynamic batches and use paged memory to serve them in parallel. As a sequence generates more tokens, it also occupies more memory.
When a request enters a loop and approaches the token limit, it consumes a disproportionate share of the available resources for too long. The scheduler has less room to add new requests to the batch. Parallelism decreases, and overall performance drops.
In tests with three datasets, the average duration of healthy requests increased by at least 15% when they shared the machine with a degenerate sequence. In one of the datasets, the increase exceeded 71%.
The healthy request had not become more difficult. The system handling it had become slower.
Why text degeneration happens
The cause is related to the most common training objective in language models: maximum likelihood. During training, the model learns to assign a high probability to the token that comes next in the reference data.
This process works very well for learning continuations. But the model does not directly evaluate whether the complete sequence it will generate is coherent or whether it will end naturally. It only tries to predict the next token at each step.
In certain regions of the probability distribution, tokens or fragments that have already appeared in the recent context can become even more likely. The model repeats part of the sequence, that repetition reinforces the probability of producing it again, and the cycle continues.
The EOS token, which would allow the response to close, is left with a very low probability compared with the repeated fragment. The result is a loop that only ends when something external intervenes, such as the token limit, a streaming interruption, or the exhaustion of the memory available to the sequence.
Research by Holtzman and his collaborators described this phenomenon in 2020. Since then, different decoding strategies have tried to reduce it: adjusting the temperature, using top-p, applying repetition penalties, or modifying beam search.
These techniques can lower the probability of entering a loop. But they do not eliminate the problematic region of the distribution that the model learned during training.
Benchmarks are not measuring this cost
The analysis raises an uncomfortable question: if degeneration can significantly reduce production performance, why does it not appear as a standard metric in model evaluations?
The likely answer is that most benchmarks focus on the average quality of responses. They measure accuracy, recognition, or similarity to a reference, but they tend to leave out pathological cases and their consequences for the system.
That can lead to incomplete comparisons. Two models may achieve very similar quality scores and still have completely different degeneration rates. The model with the slightly higher score will not necessarily be the best option for production if it consumes much more time and memory.
In the experiment, the base version of Qwen2.5-VL-7B-Instruct recorded a degeneration rate of 2.42%. Although that may seem like a small figure, it was enough to have a considerable impact on total processing time.
That is why the proposal is to treat the degeneration rate as a first-class metric, alongside:
- Output quality.
- Latency.
- Throughput, or requests processed per unit of time.
- Memory consumption and inference cost.
- Percentage of requests that reach the maximum token limit with repetition at the end.
Detecting it in real time helps, but it is not enough
A common response is to detect repetition as the text is generated, cancel the request, and try again with a different configuration or model. This strategy can protect the system from the most visible cases.
However, it has limits. The detector must run on every output, including normal ones, so it adds compute overhead and can increase latency. In addition, retries multiply the cost of affected requests.
There is another risk: not all repetition is an error. A document may contain lists, legal terms, or phrases that are repeated legitimately. An overly aggressive rule can cancel correct responses, while an overly flexible rule can let a loop through.
The inference layer can contain the problem, but it does not eliminate the computation that has already been spent or change the cause producing it. If the failure originates in the distribution learned by the model, a more complete solution must act during training.
DPO reduces loops at the model level
The work evaluated a two-stage process. First, supervised fine-tuning, or SFT, was applied using examples aligned with the OCR domain. This stage improves the model's ability to solve the task, but it does not always eliminate degenerate patterns inherited from pretraining.
Then Direct Preference Optimization, known as DPO, was used. Instead of using pairs of general responses, the team built examples where the rejected response was a degenerate generation from the model itself and the chosen response was a healthy output.
In this way, the training did not just teach the model which response to prefer. It also pushed the model away from the probabilistic regions associated with its own failures.
The results were significant: across five model families ranging from 3 billion to 7 billion parameters, DPO reduced degeneration by between 37% and 87% compared with using SFT alone.
The most notable case involved a 3-billion-parameter model, Nanonets-OCR2, whose rate fell from 1.61% to 0.20%, a reduction of 87.6%. In general models with around 7 billion parameters, the reductions ranged from 37% to 56%, with an average of 59.4% across the evaluated families.
Training history matters more than size
One of the most interesting conclusions is that the largest model was not necessarily the most stable. The smaller specialized model achieved the lowest degeneration rate among the models tested.
This suggests that stability depends less on the number of parameters and more on the distance between the model's training history and the task it must solve. Specialization can change not only average quality, but also how failures emerge.
For those deploying language models, the lesson is practical: it is not enough to ask which model achieves the best score on a benchmark. You also need to measure how many requests enter loops, how long they occupy the system, and how they affect other requests sharing the GPU.
Text degeneration is not a futuristic problem or an academic oddity. It can appear in real OCR systems, assistants, structured extraction, and content generation. If it is not tracked, it becomes an invisible cost inside the processing queue.
Measuring it requires data that servers already commonly produce. Reducing it may require better training strategies, not just decoder adjustments. And evaluating it alongside quality would enable deployment decisions much closer to operational reality.
Original source
https://huggingface.co/blog/Dharma-AI/text-degeneration-a-production-failure-mode-that-m