Language models don’t just need more data: they need data suited to each language and community. A team of Thai researchers adapted Ai2’s open Dolma toolkit to create Mangosteen, a 47-billion-token corpus designed specifically to train AI models in Thai.
The result highlights an important idea: cleaning less data—but applying better local criteria—can be more useful than accumulating enormous amounts of text from the internet.
Dolma makes it possible to adapt data curation
Dolma is a set of open-source tools for building and cleaning the large corpora used to train language models, including Ai2’s open OLMo models.
The key difference is that Ai2 didn’t publish only a finished dataset. It also released the process so other researchers could inspect, modify, and reproduce it. Why does that matter? Because rules that work for English don’t necessarily work for Thai, Spanish, or any other language.
The Mangosteen project took advantage of that flexibility. Instead of creating a curation pipeline from scratch, the team started with Dolma and focused its efforts on adapting each stage to the characteristics of Thai.
Opening up the tools allowed local researchers to adjust the technology using knowledge that external teams would have difficulty providing on their own.
The problem with Thai data
A large portion of the public datasets in Thai had been built mainly from web crawls, meaning large-scale automated collections of internet pages. The team found that these collections had not always been sufficiently reviewed by native speakers.
They could also include content that was not appropriate for training and leave out valuable sources, such as:
- Books.
- Research articles.
- Official websites.
- YouTube subtitles.
For a small team, designing a complete system for extraction, filtering, deduplication, and evaluation from scratch would have taken too much time. Dolma provided a functional foundation, but it also revealed that some of its decisions needed to change in order to work properly with Thai.
When English rules don’t work
One of the problems appeared in sentence- and paragraph-level deduplication. These techniques seek to remove repeated content so a model does not see the same text many times during training.
However, Thai does not mark sentence boundaries in the same way as English. When the team applied those rules directly, it discovered that it could eliminate almost all of its data. The solution was to keep document- and URL-level deduplication, which was useful, and redesign the processes that depended more heavily on the structure of the language.
They also replaced specific tools, adjusted the quality filters, and added rules to recognize patterns that frequently appear on Thai websites. One example involved certain news sites that contained only incomplete snippets followed by messages such as "Read More." Without an additional filter, those pages could enter the corpus as if they were complete articles.
A smaller, but more useful, corpus
The team used its adaptations to create Mangosteen and compared it with large web data sources, including Common Crawl and FineWeb2. The latter had already been cleaned and prepared for model training.
The Mangosteen pipeline removed more than 80% of the initial data from Common Crawl and nearly half of FineWeb2. Despite working with less information, the models maintained or improved their performance compared with models trained on the larger datasets.
That result challenges a widespread assumption: that a larger corpus always produces a better model. In reality, a dataset with fewer duplicates, less incomplete content, and more relevant sources can provide a more valuable training signal.
Better representation of Thai culture
The improvements also appeared in larger models. Models trained with Mangosteen achieved better performance on evaluations of Thai cultural knowledge.
This doesn’t mean the model automatically becomes perfect or understands a culture the way a person would. It does suggest that including local sources and applying criteria defined by speakers of that language can help better represent a community’s history, references, and context.
The lesson applies to many countries. A model trained mainly on global or English-language data may perform well on general tasks, but fail when it has to interpret local institutions, expressions, historical figures, or cultural references.
Why opening up the process matters
The Mangosteen case shows that transparency isn’t only about publishing a model’s weights or making a dataset available for download. It also means sharing the tools and decisions that determine which data makes it into training.
With access to Dolma’s code, the team was able to inspect the pipeline’s architecture, change its components, test new rules, and reproduce the results. That control allowed them to build a corpus aimed at the needs of Thai users, rather than accepting without modification a process designed for another context.
Multilingual AI doesn’t improve simply by adding more languages to a list. It also needs data selected with an understanding of how people write, publish, and live in each community.
Mangosteen is a concrete example of how open tools can distribute innovation. A small group didn’t have to rebuild the entire infrastructure and could invest its time in solving the problems that truly mattered for its language.
As language models expand, this ability to adapt will become increasingly important. The question won’t only be how many tokens a model has, but who selected those tokens, according to what criteria, and for whom the technology works best.