vLLM integrates a transformers backend at native speed | Keryc
The transformers library can now run inside vLLM reaching native speeds. What does that mean for you? That models published on Hugging Face can run with vLLM's optimizations without manually porting code, while keeping compatibility with training and deployment.
What changes
Update your package with pip install --upgrade vllm --torch-backend auto and then add the flag --model-impl transformers to the serve command to use any Hugging Face-compatible model inside vLLM.
This is no longer just a functional integration: the transformers backend for vLLM now applies run-time transformations that let it match or exceed the performance of hand-written native implementations in vLLM.
How they achieve it (technical)
The key is analyzing and rewriting parts of the model graph to apply fusions and mappings to optimized kernels. They use torch.fx for static graph analysis and then manipulate code with ast to rewrite operations in-place.
After the rewrite, models remain compatible with torch.compile and CUDA Graphs — so you can still benefit from low-level compilation and optimizations just like with a native implementation.
Fusions and parallelisms
Among the optimizations are fusions that group several operations into vLLM's ultra-optimized kernels, including support for Expert Parallelization in MoE models. They also identify and fuse blocks like MergedColumnParallelLinear and QKVParallelLinear, which allows inferring plans for tensor-parallel (TP) and, when possible, pipeline-parallel (PP).
That means complex parallelism setups that used to require a manual port can now be activated automatically when the model is compatible.
Training compatibility
Unlike native vLLM implementations, the transformers implementations remain usable for training, eval, and RL rollouts. In practice you can use the same model code for the whole cycle without sacrificing inference performance.
Performance and real cases
The team tested the transformers backend against vLLM native implementations on three Qwen3 variants:
Qwen3-4B dense on a single GPU
Qwen3-32B dense with tensor-parallel across 2 GPUs
Qwen3-235B FP8 Mixture-of-Experts with data + expert parallel across 8x H100
Main result: the transformers backend matches or exceeds native throughput in all three cases.
Practical tip: add --max-model-len 8192 if your node has memory constraints.
The full set of benchmarks and the reproducible runner are available as a gist script named benchmark.sh if you want to reproduce the tests.
Limits and considerations
Models that use linear attention are not supported today, though they will be soon.
Model repos with custom code on the Hub may not work if they don't follow the expected conventions.
This automated integration seeks compatible models; for very exotic models or operations that aren't recognized, a manual port can still extract more performance.
What it means for teams and developers
Want to deploy Hugging Face models at maximum performance without rewriting anything? Now it's more feasible. This speeds up the path from a public checkpoint to an optimized, scalable inference service.
For research and product teams the advantage is twofold: less friction to try new models in production and a single model codebase usable both for training and inference.
