Embedding models usually summarize each document into a single vector. That strategy is fast and practical, but it can also lose important details: a product code, a proper name, or a specific condition within a long text. Sentence Transformers aims to solve that limitation with MultiVectorEncoder, a new API for search models based on late interaction.
The proposal integrates ColBERT-style models, PyLate checkpoints, Stanford-NLP models, and visual retrieval systems such as ColPali. What’s the advantage? Instead of compressing the entire document into a single representation, it preserves one vector per token and compares the relevant parts directly.
What changes compared with a traditional embedding
A dense model converts an entire text into a fixed-size vector, for example one with 384, 768, or 1024 dimensions. Then, the search compares the query vector with each document vector using a dot product or a similar metric.
This approach works well, but it forces the model to mix everything into a single representation. Imagine a search for green sofa with wooden legs and rounded cushions. A dense embedding might also bring a green sofa with metal legs closer, because all the concepts end up combined in the same point.
Multi-vector models maintain a matrix of vectors for each document. A nine-token text can be represented as a 9 x 128 matrix instead of a single 1 x 128 vector. Each token preserves a contextualized representation and can find its best match within the document.
Late interaction preserves the speed of indexing documents separately, while allowing a much more detailed comparison during the search.
MaxSim: how relevance is calculated
The main score is called MaxSim. For each token in the query, the system looks for the document token with the highest similarity and then adds all those maximum values.
In simplified form:
MaxSim(query, document) =
sum of the maximum similarity for each query token
Because the vectors are normalized with the L2 norm, each comparison is usually equivalent to a cosine similarity between -1 and 1. The final score depends on the number of tokens in the query, so you shouldn’t directly compare values produced by models with different configurations.
You can also use MeanMaxSim, which divides the sum by the number of query tokens and produces an average score that’s easier to interpret.
The match doesn’t have to be literal. In a test with the query Where do penguins live? and the text Penguins inhabit Antarctica, the token live found a strong relationship with inhabit, even though the words don’t share characters. At the same time, an exact identifier or a proper name can preserve its own signal instead of getting diluted among all the other concepts.
Using it with Sentence Transformers 6.0
Installation keeps the library’s familiar workflow:
pip install -U sentence-transformers
Visual document retrieval requires the image dependencies:
pip install -U "sentence-transformers[image]"
Loading the model also follows a familiar API:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/LateOn")
The library can detect different checkpoint formats, including PyLate and Stanford-NLP ColBERT models:
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
model = MultiVectorEncoder("answerdotai/answerai-colbert-small-v1")
model = MultiVectorEncoder("mixedbread-ai/mxbai-edge-colbert-v0-17m")
You must use different methods for a query and a document. This isn’t a minor detail: these models often use different prefixes, length limits, and masks for each side.
queries = ["What is the capital of France?"]
documents = [
"Paris is the capital of France.",
"Berlin is the capital and largest city of Germany."
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
scores = model.similarity(query_embeddings, document_embeddings)
The result isn’t a single rectangular tensor. Each entry returns a matrix with as many vectors as the number of tokens it retains. A long document may have a taller matrix than a short one.
Two ways to take it to production
Exhaustive search is useful for small corpora. You encode the documents once and then calculate MaxSim against all of them. In a test with 4,874 Natural Questions passages, the model produced 608,414 token vectors.
The process took approximately 20 seconds on an RTX 3090, and each query took about 120 milliseconds using an exact comparison. That’s a reasonable result for thousands of documents, but the cost grows with the total number of tokens, not just the number of documents.
For large collections, there are two main alternatives:
- Native late-interaction indexes: Qdrant, Weaviate, Vespa, LanceDB, VectorChord, Milvus, and
fast-plaidoffer different ways to store and query multi-vector representations. - Retrieve and rerank: a dense model retrieves the top candidates, and the multi-vector model reranks them with
MaxSim.
The second pattern is especially practical. You can keep a conventional dense index and apply the multi-vector model only to the 50 or 100 most promising candidates. This gives you better precision without multiplying the size of the index.
The cost: more vectors, more storage
The improvement in quality comes at a price. A dense model stores one vector per document, while a multi-vector model stores one vector for each relevant token.
In the Natural Questions example, the representations occupied approximately:
| Representation | Vectors | Dimensions | Size in float32 |
|---|---|---|---|
all-MiniLM-L6-v2 | 4,874 | 384 | 7.5 MB |
gte-modernbert-base | 4,874 | 768 | 15.0 MB |
LateOn multi-vector | 608,414 | 128 | 311.5 MB |
The uncompressed multi-vector index was about 42 times larger than the MiniLM index. However, techniques such as PLAID reduce storage through centroids and quantized residuals. In the same test, the compressed index occupied about 92 MB.
The other key tool is HierarchicalTokenPooling. This technique groups nearby tokens and replaces them with an average representation. With a grouping factor of 2, the index can be reduced by approximately half. In evaluations cited by Hugging Face, the average performance loss was small, although you should always measure the effect using your own data.
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling
pooling = HierarchicalTokenPooling(pool_factor=2)
document_embeddings = model.encode_document(
documents,
token_pooling=pooling
)
Retrieving visual documents without OCR
One of the most interesting applications appears when the document isn’t plain text. ColPali-family models can compare a written query with the full image of a page, including tables, charts, layout, and visual positions.
model = MultiVectorEncoder("vidore/colqwen2.5-v0.2")
queries = [
"What variable appears on the chart’s vertical axis?",
"In which year was total spending the highest?"
]
images = [
"https://example.com/page-1.jpg",
"https://example.com/page-2.jpg"
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(images)
scores = model.similarity(query_embeddings, document_embeddings)
Here, the document vectors represent image patches, not words. The query can locate information inside a table or chart without first passing through an optical character recognition system.
This opens possibilities for financial files, scientific reports, forms, invoices, and historical documents. It also requires more memory: a page can generate hundreds of vectors, compared with the roughly 125 average vectors in the text passages used in the example.
Audio, video, and interpretability
The same idea can be extended to other modalities. vidore/colqwen-omni-v0.1 accepts text, images, audio, and video through a common interface. A text query such as medicine for motion sickness can retrieve a conversation in which the person says carsickness, without having to transcribe the audio first.
With video, you need to control frame sampling. Processing every frame consumes too much memory, so it’s better to use a lower rate, such as 0.5 frames per second, especially with long clips.
Another advantage is interpretability. Because MaxSim assigns a specific part of the score to each query token and its best match, it’s possible to explain why a document ended up in a particular position.
In images, that information can be turned into heat maps over the page. In text, you can show that preliminary matched the same term, while when found a contextual relationship with since. This isn’t a perfect explanation of the model’s reasoning, but it is precise evidence of the matches that drove the score.
Performance and evaluation
Sentence Transformers supports PyTorch, ONNX, and OpenVINO, as well as reduced precision, Flash Attention, and torch.compile. In the tests described, float16 with Flash Attention reached up to 2.44 times the performance of float32 without a measurable loss in retrieval quality.
There is one important exception: some ColBERT checkpoints use query expansion with [MASK] tokens that require attention. In those cases, Flash Attention may be incompatible, so it’s best to use SDPA.
To evaluate models, the library adds MultiVectorNanoBEIREvaluator, which lets you run a compact version of 13 BEIR datasets. In the comparison between LateOn and DenseOn, both trained with the same ModernBERT backbone and 149 million parameters, the multi-vector model achieved an average NDCG@10 of 0.6868, compared with 0.6764 for the dense model.
LateOn won on 9 of the 13 datasets, although not all of them. It lost on ArguAna, FiQA2018, SCIDOCS, and SciFact. What’s the lesson? Late interaction offers an average improvement, but it doesn’t replace evaluation with the real queries and documents from your application.
When should you use it?
A multi-vector model makes sense when relevance depends on specific details, partial matches, or several requirements at once. It’s also a strong option for long documents, specialized domains, and visual retrieval.
A practical guide would be:
- Use dense embeddings if you need simplicity, low resource consumption, and large-scale searches.
- Use multi-vector as a reranker if you already have a dense search engine and want to improve its top results.
- Use a dedicated multi-vector index when retrieval quality justifies the additional storage cost.
- Apply token pooling when the number of vectors is the main problem.
- Always evaluate on your own data, especially if you work with Spanish, technical documents, or images with unusual layouts.
Adding MultiVectorEncoder doesn’t turn every search system into ColBERT overnight. The important thing is that it reduces friction: loading, encoding, comparing, and evaluating these models now follow a unified workflow within Sentence Transformers.
Artificial intelligence isn’t inventing a magical search. It’s preserving more clues before deciding which document answers best. In many applications, that difference between summarizing too much and retaining detail may be exactly what separates a correct result from one that is merely similar.