Nunchaku arrives in Diffusers and changes how you run diffusion transformers: less memory, faster inference, and no CUDA compilation on your machine. Sounds good? Here I explain what's behind it, how to use it, and when it makes sense to quantize your models.
What is Nunchaku and why it matters
SVDQuant, the technique behind the Nunchaku engine, is not the usual "weights-only" quantization. Instead of storing weights in low precision and dequantizing them during computation, SVDQuant represents the hardest matrices with a low-rank branch at 16 bits and quantizes the rest to 4 bits.
The result: W4A4 (weights and activations in 4 bits), which reduces memory and also speeds up the denoising loop.
Until now, using Nunchaku checkpoints required a separate inference engine. With Nunchaku Lite, Diffusers can load those checkpoints directly with from_pretrained() and without local compilation: the kernels are downloaded from the Hub via the kernels package.
Nunchaku Lite: integration and execution model
How does it work inside Diffusers? Nunchaku Lite patches relevant nn.Linear modules at load time with SVDQ/AWQ layers that implement SVDQuant and AWQ. The optimized kernels come from the Hub and there are two main families:
svdq_w4a4: W4A4 with low-rank correction. Available inint4andnvfp4variants.awq_w4a16: weights in 4 bits with activations at 16 bits, useful for projections that are sensitive to precision like adaptive normalizations.
The advantage: a Nunchaku Lite repository is a normal Diffusers repo. Schedulers, LoRA, offloading and torch.compile keep working because the module structure is preserved.
Compute, compatibility and GPU variants
Nunchaku Lite picks kernels depending on the GPU and the checkpoint variant:
nvfp4(NVFP4): requires Blackwell GPUs (RTX 50, RTX PRO 6000, B200).int4: compatible with Turing / Ampere / Ada (RTX 30/40, A100, L40S).
Volta and Hopper aren't supported by the 4-bit kernels right now. The loader validates CUDA capability and fails with a clear error on unsupported GPUs.
Performance and memory: real numbers
In Hugging Face's measurement (RTX PRO 6000, 1024x1024 with rootonchair/ERNIE-Image-Turbo) they observed:
- BF16 baseline: 3.00 s, 31.1 GB VRAM.
- Nunchaku Lite NVFP4: 2.27 s (1.35x), 20.6 GB VRAM.
- Nunchaku Lite NVFP4 +
torch.compile: 1.68 s (1.8x), 20.6 GB VRAM. - Nunchaku Lite NVFP4 + NF4 text encoder: 2.29 s, 16.0 GB VRAM.
In short: up to 50% less VRAM and ~30% latency improvement; torch.compile helps close the gap with the original engine.
Why the original engine is still faster for some models?
Nunchaku (the original engine) uses architecture-specific fused kernels: fused QKV projections, fused GELU+MLP, and so on. Those optimizations require structural rewrites of the model (for example, merging to_q/to_k/to_v into a single to_qkv), which yields more speed.
Nunchaku Lite is generic: it doesn't automatically infer those rewrites. Still, it delivers a significant speed boost without per-model integration work.
How to get started (quick commands)
Install dependencies:
pip install -U diffusers transformers accelerate kernels bitsandbytes
Load a pre-quantized pipeline like any Diffusers model:
import torch
from diffusers import ErnieImagePipeline
pipe = ErnieImagePipeline.from_pretrained(
"lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder",
torch_dtype=torch.bfloat16,
).to("cuda")
image = pipe(
prompt="A cinematic portrait of a red fox in a misty forest at sunrise, detailed fur",
height=1024, width=1024, num_inference_steps=8,
guidance_scale=1.0, generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("output.png")
You don't need custom pipeline classes or to compile CUDA locally: NVFP4 kernels are downloaded the first time from the Hub.
Combinable optimizations
torch.compilecan raise the speedup from ~1.35x to ~1.8x.- Quantizing the text encoder (for example to NF4 with
bitsandbytes) reduces peak VRAM further (benchmarks showed ~22% additional reduction). - Offloading (like
enable_model_cpu_offload) works with nunchaku-lite models.
Quantizing your own models with diffuse-compressor
The full flow is available in diffuse-compressor: inspection, calibration, SVDQuant quantization, packaging and publishing.
Quick inspection example:
python examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \
--precision int4 --rank 32 --inspect-config
Quantize the transformer:
python examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \
--precision int4 \
--output outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors
Convert to a Diffusers repo ready to publish:
python examples/convert_nunchaku_lite_diffusers.py \
--checkpoint outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors \
--model-id black-forest-labs/FLUX.2-klein-4B \
--bnb4-text-encoder text_encoder \
--compute-dtype bfloat16 \
--output-dir outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder
Then load and test:
from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained(
"outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder",
device_map="cuda",
)
image = pipe("A glass robot in a greenhouse, cinematic lighting", num_inference_steps=4).images[0]
If everything looks good, pipe.push_to_hub("your-username/your-model-nunchaku-lite-int4") and others will be able to use it with from_pretrained().
Structural rewrites and when you need an adapter
The generic path assumes the model doesn't need structural rewrites. But models like FLUX.1-dev require grouping Q, K and V into a single to_qkv projection to use the fused operator of the Nunchaku engine.
That is handled with a target configuration during quantization and with small adapters at load time.
Diffusers includes examples and guides for those rewrites and adapters when they're necessary.
Ready checkpoints and resources
Some ready-to-use repositories:
rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoderrootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoderOzzyGT/Krea_2_Turbo_nunchaku_lite_nvfp4lite-infer: collections and more checkpoints
Also, diffuse-compressor streamlines the pipeline for new models.
Final thoughts
Nunchaku Lite brings SVDQuant's power into the Diffusers ecosystem in a practical way: less memory and better latencies without sacrificing pipeline interoperability. Want maximum speed and can invest work to integrate architecture-specific changes? Then the original Nunchaku engine with fused rewrites is still the top choice.
Prefer a simpler, reproducible route? Nunchaku Lite lets you quantize and publish models today.