NeoMME accelerates multimodal document search

H Company presents NeoMME, a family of multimodal and multilingual encoders designed to understand text and images with a single architecture. Its 260-million- and 800-million-parameter models aim to offer a more efficient alternative to systems that combine a separate visual encoder with a generative language model.

The proposal is especially interesting for searching through documents, PDFs, tables, and charts without first relying on an optical character recognition system, commonly known as OCR.

One Transformer for text and images

Many vision-language models work with three main components: an encoder that analyzes the image, a projector that adapts its features, and a causal language model that processes the input. This structure is useful for generating text, but it can be more expensive than necessary when the task is to classify, retrieve, or compare information.

NeoMME takes a different approach. It uses a single bidirectional Transformer that processes both text tokens and the original patches of an image. The model does not start with a visual encoder, a text encoder, or a pretrained decoder: everything is trained from scratch.

The central idea is simple: if the task is to find useful representations, you do not need to generate an answer word by word.

Images are divided into 32-by-32-pixel patches, and those fragments enter the same computational flow as the text. This makes it easier to share training, parallelization, and inference serving between both modalities.

NeoMME also supports dynamic resolution. It preserves the image’s aspect ratio and size, so a page with many details can receive more tokens than a small, simple image. Its context reaches 16,384 tokens, which, according to its creators, is enough to process up to two standard 4K UHD images measuring 3840 by 2160 pixels.

Multilingual training from scratch

The family includes a BPE tokenizer with 131,000 tokens, trained on multilingual text, code, mathematics, and transcriptions generated from images. The goal is for the encoder to work with different languages and types of content instead of being limited to documents in English.

Pretraining uses a masked discrete diffusion task. In practical terms, the model receives text with some tokens hidden and must reconstruct them using the available context. In multimodal examples, the images remain visible while the system tries to recover the deleted text.

With light masking, the model can complete a sentence based only on nearby words. But when a larger proportion is hidden, it needs to rely on the image to understand what appears in the document and produce a coherent description.

Training combined text, code, mathematics, natural images, and visual documents. Each model processed approximately 524 billion packed tokens, including 290 billion from exclusively textual examples.

NeoMME-Retriever searches pages, not text fragments

The team adapted NeoMME for visual document retrieval using an approach inspired by ColPali. Instead of converting a PDF into text fragments, the system turns each page into an image and searches directly across it.

This detail matters. An image preserves the layout, tables, charts, font sizes, and spatial relationships between elements. Even highly accurate OCR can lose some of that information when it converts a complex page into plain text.

NeoMME-Retriever produces two types of representations in a single pass:

• Dense embeddings: summarize the entire input in a normalized vector. They are compact and work well with approximate nearest-neighbor indexes, such as those used in vector databases.
• Late-interaction embeddings: retain separate vectors for each text token or image patch. This allows them to detect local matches between a query and specific regions of a page.

Late interaction usually offers greater precision, although it requires storing more vectors. In very large corpora, a practical strategy is to use dense embeddings first to retrieve a small number of candidates and then apply late interaction to rerank them.

Results in visual retrieval

In tests on ViDoRe v3, NeoMME-Retriever-260M achieved an nDCG@10 of 0.523. According to H Company, it was the highest-scoring model among those evaluated with fewer than 800 million parameters and fell just 0.002 points short of ColQwen2.5, which uses around 14 times more parameters.

The 800-million-parameter version reached 0.556, less than 0.009 points behind Vultron Retriever Flash, a similarly sized model. Both versions sat on the benchmark’s Pareto frontier, which compares model quality and size.

The results should be interpreted within the evaluation conditions and the models being compared, but they point to a relevant conclusion: a smaller multimodal encoder can come close to the performance of much larger systems on retrieval tasks.

Less storage and greater speed

Late interaction has an obvious cost: a 2048-by-2048-pixel page generates approximately 4,200 vectors. In float32 format, that equals about 2.1 MB per page, with an average close to 1.5 MB in ViDoRe v3.

NeoMME reduces this consumption with two techniques. The first is hierarchical token clustering, which combines similar vectors and replaces them with their average. The second is asymmetric quantization, which converts document vectors into more compact formats, such as int8 or binary, while keeping queries at higher precision.

With clustering and int8 quantization, storage dropped to about 39 kB per page—a 39-fold reduction—while retaining more than 99% of the original nDCG@10. In a more aggressive configuration, it reached 6 kB per page, 255 times less, while preserving more than 95% of retrieval quality.

Speed is another highlight. On an NVIDIA L40S GPU and with 2048-by-2048-pixel images, the 260M model encoded about 51 pages per second, compared with 26 pages per second for ColModernVBERT under the same conditions.

Available for building visual RAG

NeoMME checkpoints are released under the Apache 2.0 license and are available through Hugging Face Transformers. There is also a demo on Hugging Face Spaces and support for fine-tuning the models with Sentence Transformers 6.

One direct application is visual retrieval-augmented generation, or visual RAG. The workflow consists of converting each page of a PDF into an image, generating its embeddings, storing them in a vector database such as Qdrant, Weaviate, or Milvus, and retrieving the most relevant pages for a query.

Those images are then sent to a vision-language model that generates the answer. What is the advantage? The system can directly consult tables, diagrams, charts, and visual layouts that traditional text extraction might simplify or lose.

NeoMME is not trying to replace every generative model. Its contribution is showing that many multimodal tasks need a good representation system, not necessarily a model that generates text autoregressively. For companies indexing large volumes of documents, that difference can translate into less memory, lower inference costs, and faster searches.

Original source

[https://huggingface.co/blog/Hcompany/neomme]