Generating text with a language model may seem instantaneous, but behind every response there is a fairly methodical process: the model produces one token, then another, and then another. NVIDIA wants to change that dynamic with Nemotron-Labs Diffusion, a family of models that can generate multiple tokens in parallel, review them, and combine them with traditional autoregressive generation.
From token by token to generating and refining
Autoregressive language models, like many of the ones we use every day, write from left to right. Each new token depends on the previous ones and requires a new complete pass through the model. This approach is stable, well understood, and has driven much of the recent progress in artificial intelligence.
The problem appears when low latency is required, individual queries are being handled, or you are trying to make better use of a modern GPU. In those scenarios, a considerable portion of the time may be spent moving weights from memory instead of performing useful calculations.
Also, once an autoregressive model generates a token, it usually cannot review it naturally. If it makes an early mistake, that error can influence everything it writes afterward. What would happen if the model could draft an entire block and then correct it?
Diffusion language models, known as DLMs, follow precisely that idea. They generate multiple tokens in parallel and refine them over several iterations. Reducing the number of iterations lowers computational usage, while increasing the steps can favor a more elaborate response.
Three ways to generate with the same model
Nemotron-Labs Diffusion does not force you to choose between an autoregressive model and a diffusion model. The same family supports three generation modes, selectable during deployment:
- Autoregressive mode: works like a conventional causal model, generating text from left to right. It serves as a correctness reference and maintains compatibility with existing workflows.
- Diffusion mode: generates blocks of tokens and progressively refines them. In the implementation described by NVIDIA, the model works with blocks of 32 tokens and decides which ones are sufficiently consolidated using a confidence threshold.
- Self-speculation: the model creates several candidate tokens in parallel and then verifies them autoregressively. This combines the speed of a parallel proposal with the reliability of checking tokens one by one.
This flexibility is important for development teams. Switching from one mode to another does not require redesigning the entire application; you only need to adjust a configuration option during inference. The same application could prioritize accuracy, speed, or a balance between the two depending on the workload.
What happens to accuracy?
Speeding up generation is not very useful if quality drops significantly. According to NVIDIA’s evaluations, Nemotron-Labs Diffusion 8B achieved an average improvement of 1.2% over Qwen3 8B on the tasks analyzed.
For decoding efficiency, NVIDIA uses a metric called tokens per forward pass, or TPF. Unlike a measurement tied to specific hardware, this metric aims to show how many tokens the model processes during each computational run.
The published results indicate that:
- Diffusion mode achieved up to 2.6 times more TPF than the autoregressive models used for comparison.
- Self-speculation reached 6 times more TPF with linear speculation.
- Quadratic speculation reached 6.4 times more TPF.
- In a test with an NVIDIA B200 GPU, LinearSpec achieved nearly 865 tokens per second, approximately four times the reference autoregressive performance on the same hardware.
These figures should be understood as results from NVIDIA’s tests, not as a universal guarantee. Final performance will depend on the hardware, context length, batch size, inference configuration, and specific task.
The architecture combines the autoregressive past with diffusion
Text diffusion models have faced several obstacles: lower accuracy than the best autoregressive models, more complex training, and limited compatibility with attention caches, known as the KV cache.
The KV cache stores some of the calculations that have already been performed to avoid repeating them during generation. It is a fundamental component for accelerating autoregressive inference, so any alternative that cannot take advantage of it starts with a practical disadvantage.
Nemotron-Labs Diffusion adopts a hybrid strategy. NVIDIA starts with an autoregressive model and adds diffusion capabilities through additional training. The goal is to preserve the skills learned during the original training while also enabling parallel generation by blocks.
Training used a combined autoregressive and diffusion objective. The pretraining phase used 1.3 trillion tokens from NVIDIA Nemotron’s pretraining datasets. The model then received supervised fine-tuning with another 45 billion tokens from the company’s post-training datasets.
This is not, therefore, about abandoning traditional generation completely. The proposal is to turn both approaches into capabilities available within the same model, as if a developer could choose between a steady stream, parallel drafting, or a combination of the two.
Available models and possibilities for developers
The family includes text models with 3 billion, 8 billion, and 14 billion parameters, as well as an 8-billion-parameter vision-language model. NVIDIA offers base variants and versions fine-tuned to follow instructions and hold conversations.
The text models are released under the NVIDIA Nemotron Open Model License, designed to facilitate commercial use. The vision-language model uses the NVIDIA Source Code License, which NVIDIA presents as an option with broad flexibility for research. Before integrating any of them into a product, it is worth reviewing the exact terms of each license.
NVIDIA is also releasing the training code through the Megatron Bridge framework. Integration with SGLang is planned for its main branch, while inference support is initially available through a request in the project repository.
In practice, the configuration can define whether the model works in autoregressive, diffusion, or self-speculation mode. In autoregressive mode, you can enable ar_mode=true. For the other modes, NVIDIA mentions implementations such as FastDiffuser and LinearSpec, focused respectively on diffusion performance and speculative verification.
Why this proposal matters
Parallel generation can be especially useful in coding assistants, document editing, autocomplete, translation, and applications where a fast response has a direct impact on the user experience. It can also help with infilling tasks in the middle of a text, because the model is not limited to completing only from the last token.
The ability to review tokens introduces another important difference. A traditional autoregressive model writes like someone who cannot erase what they have just put on the page. A diffusion model can create a draft, detect which parts have lower confidence, and refine them before finalizing the result.
Nemotron-Labs Diffusion is not betting on replacing autoregressive generation in every situation, but on offering a single foundation with different modes for speed, review, and verification.
For developers, that compatibility may be more important than an abstract promise of speed. If the same checkpoint works as an autoregressive model, a diffusion model, and a self-speculation system, experimenting with performance requires fewer changes to the application and infrastructure.
Nemotron-Labs Diffusion shows where language inference may be headed: models capable of drafting in parallel, correcting parts of their output, and turning to autoregressive generation when accuracy demands it. The idea of generation at the speed of light remains a metaphor, but reducing the time a GPU spends waiting on memory is a very concrete goal.