Just weeks ago Google released Gemma 4, their most capable open model family, and it's already reached millions of downloads. Now the practical news: the new Multi-Token Prediction (MTP) drafters promise to speed up inference up to 3x without losing quality or logical consistency in responses.
What problem are they solving?
Have you ever had an app take forever to reply right when you need it most? That happens because large models spend a lot of time shuttling parameters between memory and processor to generate a single token. The CPU or GPU ends up waiting, underused, and latency skyrockets—especially on consumer hardware.
Developers see this as the bottleneck for putting models into production or running powerful assistants locally. The consequence? Less fluid experiences and fewer apps that actually work well on the edge or on your laptop.
What is speculative decoding and what does MTP do?
Speculative decoding separates the proposal of tokens from their verification. In plain terms: a lightweight model (the MTP drafter) suggests several tokens at once, using compute time that would otherwise sit idle, and the main model (for example Gemma 4 31B) verifies those suggestions in parallel.
If the main model agrees with the proposal, it accepts the whole sequence in one pass and even produces an extra token. The result? You can get a full sequence in the time it would normally take to produce a single token.
Key advantage: more speed without degrading accuracy or reasoning, because the final check is always done by the big model.
Practical benefits for developers and users
- Better response times: chats that feel almost real-time, smoother voice assistants, and agents that plan several steps without pausing.
- Stronger local development: running Gemma 4 26B or 31B on personal computers and consumer GPUs becomes more viable thanks to the speed-up.
- Improved edge performance: mobile and embedded devices produce outputs faster and often use less battery in many scenarios.
- Quality intact: the final verification ensures you don't lose the reasoning ability Gemma 4 provides.
A simple look at how they pull it off (without all the math)
For MTP to be fast and accurate, drafters share information with the main model: they reuse activations and the keys-and-values cache (KV cache), avoiding recomputing context that's already been processed. In edge models like E2B and E4B, they also applied tricks to speed up the embedding step when that was the bottleneck.
There are also hardware-specific optimizations. For example, on Apple Silicon some models perform better when you process multiple requests at once (batch sizes of 4 to 8), and similar gains appear on Nvidia A100 with batching. All this adds up to those ~2x to 3x improvements in practical conditions.
How can you try it today?
The MTP drafters for Gemma 4 are already available under the Apache 2.0 license. You can download weights and documentation on platforms like Hugging Face or Kaggle, and start integrating them with popular tools and runtimes (transformers, MLX, VLLM, SGLang, Ollama). There are also demos and Android/iOS packages in the Google AI Edge Gallery.
If you're a developer, checking the official docs and the technical explainer will give you the details to connect a drafter to your target model and take advantage of shared cache and other optimizations.
Quick reflection
This isn't a cosmetic change: MTP takes a research idea and turns it into a practical improvement that reduces real latency in real-world apps. If you work on assistants, autonomous agents, or experiences that need immediate interactivity, it's worth testing these drafters. Can you imagine your app responding three times faster without losing intelligence? It's available now.
Original source
https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4