Benchmarks don’t just measure artificial intelligence: they also decide which models receive attention, investment, and improvements. So when a language doesn’t appear in an evaluation table, its problems can remain off the map. The Open ASR Leaderboard aims to address that gap with new test sets for Indian English and Hindi.
A leaderboard that is starting to look toward the Global South
The Open ASR Leaderboard, focused on automatic speech recognition, has added the Monsoon en-IN and Monsoon hi-IN datasets. Hindi is the first Indic language to reach a multilingual tab that until now was made up mainly of European languages.
The news isn’t simply about adding more hours of audio. The goal is to measure how models perform with voices, accents, devices, and contexts that are often underrepresented in traditional evaluations.
If a benchmark only records what was said, but not who said it or under what conditions, it can hide important differences between communities.
All four datasets include a public split, available for anyone to evaluate their model, and a private split, reserved to limit benchmark-specific optimization. Together, they include 4,888 speakers distributed across hundreds of districts in India, with 12 attributes recorded for each person.
Why a single WER doesn’t tell the whole story
The most widely used metric in automatic speech recognition is WER, or word error rate. In simple terms, it compares a model’s generated transcript with a human reference and calculates how many words were substituted, deleted, or added.
The problem is that an average can hide inequalities. Previous research has found that some commercial systems make approximately twice as many errors with Black speakers as with white speakers. Differences related to gender, age, and accent have also been documented.
The leaderboard wasn’t intentionally hiding those differences. The problem was more basic: its test sets didn’t include enough information about the people who were speaking. If the audio doesn’t vary by region, age, or device, the evaluation can’t show how those variables affect the outcome either.
Monsoon was designed to introduce that variation. Its nine main dimensions are:
- Geography.
- Age.
- Gender.
- Vocabulary.
- Device used.
- Acoustic environment.
- Type of speech.
- Speech rate.
- The existence of multiple valid transcriptions for the same audio.
This means that two models with nearly identical overall WERs can reveal very different behaviors when analyzed by region or speaker group.
How the Monsoon datasets were built
The recordings come from spontaneous conversations between two people. Each participant used their own phone and Internet connection, indoors or outdoors. This preserves common usage conditions, including low-cost devices, ambient noise, and unstable connections.
The conversations were guided by everyday topics related to travel, health, agriculture, education, and digital services. The questions were designed to prompt stories, opinions, and memories without turning the answers into a memorized script.
The topics were proposed with the help of language models and later reviewed and localized by native linguists. Before recording, participants took a language proficiency test, received compensation, and gave consent for their data to be used for training and distribution.
Each recording went through quality checks. The language was verified, the participant’s reported gender was cross-checked, played or prerecorded audio was detected, and signals whose quality made the speech impossible to understand were discarded. Natural ambient noise, on the other hand, was preserved to keep the material realistic.
The audio was then segmented by voice activity. Each clip contains a single person and a single channel, with an approximate maximum duration of 15 seconds. The transcriptions were corrected and verified by native linguists through several independent rounds.
Four datasets for two language varieties
The data show an important difference between Indian English and Hindi:
- Monsoon en-IN public: 5.62 hours, 1,444 speakers, and 428 districts.
- Monsoon en-IN private: 5.58 hours, 1,405 speakers, and 420 districts.
- Monsoon hi-IN public: 1.33 hours, 468 speakers, and 202 districts.
- Monsoon hi-IN private: 4.47 hours, 1,571 speakers, and 295 districts.
Indian English covers the country’s six major regions. In the public split, 35% of the segments come from speakers in the south, 18% from the east, another 18% from the central region, 16% from the north, and 11% from the west.
The Hindi datasets are more concentrated in the Hindi Belt, as would be expected from the distribution of its speakers. Uttar Pradesh accounts for approximately 40% of participants, while overall coverage still includes numerous districts and states.
The diversity also appears in the devices. Each subset includes between 315 and 582 different phone models, with no single model accounting for more than 2.1% of the segments. This reduces the risk of a system appearing good simply because it was tested with a specific microphone.
More than half of the speakers appear only once. The ten participants with the most audio account for between 2.8% and 6.8% of the total duration, depending on the subset. The goal is to represent many different voices, not long sessions from a few contributors.
Metadata makes it possible to find failures that used to disappear
Each segment includes 18 fields, 12 of which correspond to metadata. These include audio ID, duration, language, transcription, age, gender, occupation, education, marital status, income range, district of origin, state, current city, years in the district, and the device manufacturer and model.
With this information, an evaluation can move from a general question such as “which model has the lowest WER?” to much more useful questions:
- Which model works best with older speakers?
- Which system maintains its accuracy on low-cost phones?
- Which model has more difficulty with certain regions?
- Does performance change depending on speech rate?
- Is a model robust to noise, or does it only work well on clean recordings?
A test conducted with the public Indian English set shows why this analysis matters. Eight models on the leaderboard achieved results between 4.81 and 4.99 WER. In the overall ranking, the difference looked minimal.
But when speakers were grouped by geographic region, the picture changed. openai/whisper-large-v3-turbo varied by 0.46 points between regions. mistralai/Voxtral-Mini-3B-2507, which was only 0.14 points behind in the average, showed a variation of 1.68 points, from 4.38 in the central region to 6.06 in the east.
In other words, two models that are almost indistinguishable in the overall table can depend very differently on where the person speaking is from. In addition, the most difficult region wasn’t the same for every system. This suggests that the problem isn’t only audio quality, but also how each model processes certain accents and linguistic patterns.
Hindi needs a different metric
In English, many spelling differences can be handled with a normalizer. For example, certain British and American variants, capitalization, punctuation, or numbers written as digits and words can be treated as equivalent.
In Hindi, the situation is more complex. Everyday speech mixes Hindi and English, many English-origin words don’t have a single accepted spelling in Devanagari, and compound words can appear together or separately. The same sentence can have numerous valid written forms.
If a single transcription is used as the reference, WER may reward a system for reproducing the spelling chosen by the annotator, even though another system correctly recognized the audio and simply chose a different variant.
That’s why Monsoon hi-IN uses a reference lattice. For each segment of the transcription, it includes a list of accepted written forms. The variants are proposed using several automatic transcriptions and language models, but they are filtered and approved by native linguists to ensure that they match what was actually said.
With this structure, OIWER, short for Orthographically-Informed Word Error Rate, is calculated. The metric aligns the model’s hypothesis with the accepted forms and penalizes only genuine recognition errors, not the choice between valid spellings.
The team also published voi-oiwer, an open-source implementation that makes it possible to reproduce the results. When comparing a single reference with the lattice, errors increased for all models, but not by the same proportion. As a result, the ranking of some systems changed.
What changes for people developing voice models
Indian English was added to the leaderboard’s main column as Voice Arena Monsoon. This means it contributes to the overall WER average for all models instead of remaining hidden behind an additional option.
The private set also feeds the aggregated private-conversation column alongside data from Appen and DataoceanAI. Hindi appears in the multilingual tab. There, a model is ranked only if it is compatible with all selected languages, enabling more balanced comparisons.
To evaluate a model on the private splits, developers must add it to the Open ASR Leaderboard through a change request in its GitHub repository. They must first report results on the public sets. The Hugging Face team then verifies those results, runs the private evaluation, and requests confirmation of the results obtained.
The value of Monsoon isn’t in declaring that a model is “good” or “bad” with a single number. It lies in making visible where it works, where it fails, and for whom. That distinction is especially important when the technology is used to transcribe calls, provide public services, create captions, or support educational tools.
A more diverse leaderboard won’t solve speech recognition bias on its own. But it changes the conversation: it allows problems involving regions, accents, devices, and writing systems to appear in the data instead of disappearing behind an apparently precise average.
Original source
https://huggingface.co/blog/open-asr-leaderboard-global-south