NVIDIA Magpie TTS speeds up multilingual voice agents

Voice agents have a very limited latency budget. Before a person hears a response, the system must capture the audio, transcribe it, process the request with a language model, and convert the final text into speech. If the last step takes too long, the entire conversation feels slow.

NVIDIA aims to solve that problem with Magpie Multilingual TTS, an open-weight text-to-speech model with support for 12 languages and deployment options within each company’s own infrastructure.

Magpie TTS bets on end-to-end control

Integrated voice services are easy to connect: you send audio and receive audio. But that simplicity also limits control over latency, privacy, customization, and model selection.

Magpie proposes a cascaded architecture. In it, each component performs a specific function and can be adjusted separately:

• A speech recognition model converts speech into text.
• A language model interprets the request and generates a response.
• Magpie transforms that written response into speech.
• NVIDIA NIM serves the models with containers optimized for GPUs.

Why does this matter? Because a company can run everything within its own infrastructure, including private environments or systems isolated from the internet. It can also measure where each delay occurs and adapt the system to its actual workload.

12 languages in a single model

Magpie TTS Multilingual has 364 million parameters and offers male and female voices in a shared multilingual representation. The new version adds three more languages:

• English
• Spanish
• French
• German
• Italian
• Vietnamese
• Mandarin
• Hindi
• Japanese
• Modern Standard Arabic
• Korean
• Brazilian Portuguese

The update also expands support for switching languages within the same conversation, especially in Hindi and Japanese. To achieve this, it uses text-to-phoneme processing through IPA and custom pronunciation dictionaries.

In practice, this can help pronounce proper names, technical terms, brands, and phrases that mix several languages correctly. Imagine a support assistant speaking in Spanish but needing to mention a product name in English without distorting it. That kind of detail can make the difference between a convincing voice and an artificial-sounding experience.

The key metric: time to first audio

In a conversational agent, it is not enough to generate all the audio quickly. It also matters how long it takes for the first fragment of speech to reach the user. This measure is known as Time to First Audio, or TTFA.

A low TTFA allows the agent to start responding while it continues generating the rest of the sentence. This reduces the feeling of waiting and brings the conversation closer to the natural rhythm of a phone call.

According to measurements published by NVIDIA for its NIM service on its own infrastructure, Magpie achieved these results:

RTFX indicates how many times faster than real time the system can generate audio. For example, a figure of 319.81x means it can produce audio at a much higher speed than normal playback, even with 64 simultaneous streams.

On a B200, the 32 milliseconds of TTFA leave more room for speech recognition and the language model to use the rest of the latency budget. NVIDIA suggests that this performance could help maintain an end-to-end conversational experience below 200 milliseconds, although the final result will depend on all components and the infrastructure being used.

These figures correspond to the optimized NIM container running on in-house equipment. The checkpoint available on Hugging Face is intended as a basis for researching and adjusting the model, while NIM is designed for production serving.

How it speeds up speech generation

Magpie’s speed does not depend solely on using powerful GPUs. The model incorporates two architectural decisions to reduce inference time without sacrificing too much quality.

The first is frame stacking. Instead of predicting one audio frame at each decoding step, the system predicts two. This cuts approximately in half the number of iterations needed to generate a response.

The problem is that producing several audio codes at the same time can create dependencies and affect quality. To compensate, Magpie uses a local transformer, which models the relationships between the generated codes and refines the result.

The combination aims to deliver two benefits at once: lower latency and a natural voice. The architecture is described in greater detail in the paper Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation, presented at ICASSP 2026.

Quality improvements in Spanish and French

The new version does more than expand language coverage. It also improves the metrics for several languages that were already available.

The Character Error Rate, or CER, measures errors in the generated text, and lower is better. The SSIM metric, used here as a reference for speaker similarity, aims for higher values.

Spanish shows a reduction in CER from 1.14% to 0.60% and an increase in SSIM from 0.715 to 0.793. In French, CER drops from 2.70% to 1.54%.

The new Modern Standard Arabic, Korean, and Brazilian Portuguese models start with CER values of 1.62%, 2.69%, and 2.91%, respectively. As always, automated metrics are only part of the story. Naturalness must also be evaluated by listening to samples in real-world situations.

Open weights for companies and developers

Magpie’s proposal is especially interesting for organizations that do not want to depend entirely on an external API. With the open weights and NVIDIA’s tools, teams can:

• Run the model on their own infrastructure.
• Keep conversations and sensitive data within the organization.
• Customize pronunciations, voices, and specialized vocabulary.
• Adjust performance according to the available hardware.
• Measure real latency without adding the round trip to a managed service.
• Work in private environments or without an external connection.

The model is distributed under the NVIDIA Open Model License. To customize it, NVIDIA recommends using NeMo, its toolkit for training and fine-tuning speech models.

The recommended inference configuration includes cfg_scale = 2.5, temperature = 0.6, top_k = 80, apply_attention_prior = True, and prior_epsilon = 0.1. These parameters help control aspects such as text fidelity, voice variability, and the use of an attention prior during generation.

From a TTS model to a complete voice agent

Magpie is not intended as a standalone solution. NVIDIA integrates it into the Nemotron Voice Agent development example, a reference architecture for building complete voice agents.

The set can combine:

• Nemotron Speech for real-time speech recognition.
• Magpie TTS for multilingual synthesis.
• Nemotron language and multimodal models for reasoning, visual understanding, and tool use.
• NVIDIA NIM for optimized inference on GPUs.
• NeMo for fine-tuning and customization.

The example includes conversations with interruptions, agents capable of interpreting images, multi-agent orchestration, tool calls, and multilingual support. The idea is to keep developers from having to assemble every piece from scratch.

This can be useful for customer service, medical assistants, enterprise copilots, translation, retail automation, and conversational applications that need to respond quickly while keeping data under control.

Magpie’s real bet is not simply that a voice will sound better. It is that teams can decide where it runs, how it is measured, which data is used for customization, and which model replaces each component. For a demo, a managed API is often enough. For a mission-critical operation, understanding and controlling the entire chain can be much more important.

Original source

https://huggingface.co/blog/nvidia/magpie-tts-multilingual-voice-agents