Can a search model understand a document better if it doesn't summarize it into a single vector? The new Sentence Transformers proposal says yes. Its MultiVectorEncoder component makes it possible to train multi-vector embedding models, also known as ColBERT or late-interaction models, and adapt them to specific domains such as medicine, law, code, or enterprise documentation.
The guide published on Hugging Face doesn't stop at theory: it explains how to choose the model, prepare the data, select the loss function, evaluate the results, and train the entire system with a single consumer GPU.
One vector per document versus one vector per token
Traditional dense models turn each complete document into a single vector. They then compare the query vector with the document vector using a similarity operation. It's an efficient strategy, but it forces the model to compress too much information into a single representation.
Multi-vector models take a different approach. They generate a small vector for each token in the text and compare the query with the document token by token. The MaxSim operator looks for the best matching element for each query token and then adds up those matches.
Instead of asking whether two texts are generally similar, the model can identify which specific parts of the query find support in the document.
This level of granularity often improves information retrieval, especially when specific terms matter. The downside is that the index stores many more vectors and needs an appropriate compression strategy.
Why adapting the model to the domain is worthwhile
A model trained for web searches doesn't necessarily understand a medical query, a legal clause, or a question about a company's internal code in the same way. Vocabulary, query style, and the definition of relevance change depending on the context.
Fine-tuning with data from the target domain allows the model to learn those signals. In the case presented by Hugging Face, medical documents averaged 941 tokens. Many available models limit documents to between 180 and 512 tokens, so they discard a significant part of the content before calculating relevance.
In that evaluation, keeping the document length limit produced differences of up to 0.24 points in NDCG@10, a common metric for measuring the quality of the first results. The conclusion is straightforward: before comparing architectures, you need to confirm that the model can read the entire document.
The training recipe
The entire workflow can be run by installing the Sentence Transformers training package:
pip install -U "sentence-transformers[train]"
The process brings together six main pieces:
- Model: an existing multi-vector checkpoint or an architecture created from scratch.
- Dataset: query and relevant-document pairs, along with evaluation data.
- Loss function: the mechanism that guides weight updates.
- Training arguments: parameters for memory, speed, precision, and tracking.
- Evaluator: a tool for measuring retrieval quality.
- Trainer: the class that coordinates all the components.
Which model to use as a starting point
One of the guide's most interesting findings appears when comparing different starting points. Checkpoints that were already pretrained for late interaction, but had not yet received final supervised fine-tuning, adapted better to the medical domain.
The lightonai/mLateOn-unsupervised model went from an NDCG@10 of 0.9087 to 0.9398 after training on 25,000 medical pairs. By contrast, some finished checkpoints declined after fine-tuning because their general training had already established patterns that didn't fit the new domain as well.
The practical recommendation is to start, in this order, with:
- A pretrained multi-vector checkpoint that has not been supervised-fine-tuned for general search.
- A strong base model with a new token-level projection added to it.
- A checkpoint finalized for general retrieval, only if the previous options aren't available.
You can also use a base transformer such as answerdotai/ModernBERT-base. Sentence Transformers adds a token-by-token projection layer, which must be trained before the model becomes useful.
Data, loss, and GPU memory
For the common case of questions and relevant passages, the guide recommends MultiVectorMultipleNegativesRankingLoss. In each batch, the correct document for a query acts as the positive, while the documents from the other queries act as negatives.
Large batches usually provide more negatives and improve learning. However, multi-vector models use a considerable amount of memory, especially when working with long documents. That's why the guide recommends CachedMultiVectorMultipleNegativesRankingLoss, a variant that makes it possible to maintain a large effective batch while processing documents in small chunks.
In the example presented, the effective batch size was 128 samples, while documents were encoded in groups of 16 to control memory usage. With documents of highly variable lengths, you can also use a maximum token budget per chunk.
There's an important technical detail: the multi-vector loss uses scale=1.0 by default. You shouldn't automatically copy the 20.0 value commonly used in some dense models. MaxSim similarity adds token-by-token matches and already produces values across a wider range. An excessively high scale can saturate the softmax and weaken the gradients.
A configuration designed for long documents
The reference training used several decisions that were tested through experiments:
- One million medical question-and-passage pairs.
- One training epoch.
- A learning rate of
1e-4. - A maximum model length of up to 8192 tokens.
[Q]markers for questions and[D]for documents.- Duplicate-free batches to improve in-batch negatives.
BF16precision on a compatible GPU.- Periodic evaluation during training.
The model also excluded punctuation tokens during comparison and storage. This modification reduced the index size by 9.6% on the evaluated set, with a moderate quality improvement.
Complete training took 14.5 hours on an RTX 3090 and used approximately 17.5 GB of video memory at most. For smaller budgets, 100,000 pairs reached results just 0.012 NDCG@10 points below training with one million examples.
Evaluate with a corpus that truly challenges the model
A common mistake is evaluating a search system against a dataset that's too easy. If each query is compared only with its correct document and a few candidates, almost any model can achieve a high score.
The guide uses 1,000 medical questions and a corpus of 200,000 passages. Relevant documents are mixed with 190,000 distractors from the training set. This allows the evaluator to distinguish genuinely useful models from models that merely take advantage of simple matches.
Sentence Transformers includes specific evaluators such as:
MultiVectorInformationRetrievalEvaluatorfor queries, corpus documents, and relevant documents.MultiVectorNanoBEIREvaluatorfor standard tests without preparing your own dataset.MultiVectorTripletEvaluatorfor query, positive, and negative triplets.MultiVectorRerankingEvaluatorfor evaluating candidate lists.MultiVectorDistillationEvaluatorfor comparing the model with a teacher.
For a real-world project, a retrieval evaluator built with data separate from the training set is usually the most useful reference. What good is a spectacular result if the test corpus doesn't resemble the searches your users will actually perform?
The fine-tuned model outperforms general alternatives
In the medical evaluation, multi-vector-encoder/mLateOn-medical achieved an NDCG@10 of 0.9139. It outperformed general late-interaction models, dense models, sparse systems, and lexical search with BM25.
Some comparative figures were:
mLateOn-medical: 0.9139.lightonai/mLateOn: 0.8520.GTE-ModernColBERT-v1: 0.8502.Qwen3-Embedding-4B: 0.7817.voyage-4-nano: 0.7563.- BM25: 0.7501.
splade-v3: 0.6853.
The fine-tuned model returned the correct result in the first position for 84.9% of queries, compared with 75.8% for the strongest general model in that comparison. That's a reduction of more than one-third in first-result errors.
These numbers don't mean that the medical model is the best choice for every task. They mean something more useful: a specialized model can outperform much larger alternatives when it understands the vocabulary and structure of its own domain.
Index size is no longer an insurmountable obstacle
Storing one vector per token may seem too expensive. In the experiment, uncompressed embeddings took up about 45 GB for 200,000 medical documents, while a dense model would have needed less than 1 GB.
But that isn't necessarily the final production size. The guide combines vector reduction, quantization, and pruning. With one-bit residual quantization using PLAID, the index dropped to 3.37 GB, with a loss of just 0.0155 NDCG@10 points compared with the uncompressed embeddings.
With additional pruning, the index reached 1.45 GB and retained an NDCG@10 of 0.8642. HierarchicalTokenPooling was also evaluated; it groups vectors from a document and stores their averages. Reducing the number of vectors by half cost just 0.0033 points and did not change first-position accuracy.
The lesson for a real implementation is clear: choosing a checkpoint isn't enough. The indexing, quantization, and compression strategy can determine both the system's cost and latency.
What this means for your own projects
The new MultiVectorEncoder API makes training ColBERT models more accessible. You don't need to start with a GPU cluster or a specialized teacher: a set of query-document pairs, a well-designed evaluation, and a consumer GPU may be enough to achieve significant improvements.
If you work with long documents or highly specific information, it's worth testing this approach alongside a simple baseline such as BM25 and a general dense model. The comparison will tell you whether token-level matches provide real value in your particular case.
Retrieval AI doesn't have to be a black box or a bet based on model size. With your own data, honest metrics, and a configuration that respects document length, you can build a specialized search engine that better understands what your audience actually needs.