要点
- Token pooling merges similar token vectors in multi-vector retrievers such as ColBERT-style models, cutting the number of stored document vectors at some cost to quality.
- At pool_factor=4, a normally trained ModernBERT-base model retained 94.6% of its unpooled NanoBEIR nDCG@10 with about 74.8% fewer document vectors. Pooling-aware training raised retention to 97.8%.
- Mean nDCG@10 at pool_factor=4 was 49.03 for the ordinary model and 51.23 for the pooling-aware model, a gain of 2.20 points. Gains were larger at tighter budgets, including +5.78 points at a cap of 4 vectors per document.
- Both models were trained from answerdotai/ModernBERT-base with a fresh 128-dimensional projection, one epoch on 120,000 MS MARCO triplets, seed 42, and no hyperparameter search. Only the loss differed.
- The method extends earlier pooling-aware training work by Stefan Josef and by LightOn. Weights are published as tomaarsen/mve-ModernBERT-base-token-pooling, with a baseline counterpart, and the recipe is implemented in Sentence Transformers.
On 11 November 2026, a Hugging Face blog post walked through a practical problem in multi-vector search. Models in the ColBERT family keep a vector for many tokens in each document, which makes matching more precise and also makes indexes large. A common fix is token pooling: merge similar token vectors so a document is stored as fewer vectors. The usual cost is a drop in retrieval quality. The post's claim is that most of that drop is a training problem. If the encoder is trained to expect its vectors to be merged, it keeps more of its unpooled quality at the same storage budget.
[1][2]What was trained, and how pooling works
A single-vector model compresses a whole document into one embedding. A multi-vector model keeps a set of contextual token vectors, so different parts of a query can match different parts of a document. The scoring often used is MaxSim: each query vector picks its best matching document vector, and those scores are added. The storage cost adds up quickly. The post gives a worked arithmetic example. At 4 bits per residual coordinate, 200 vectors of 128 dimensions take 12.8 KB per document, or 12.8 GB across a million documents. Pooling each document to 50 vectors would cut that to 3.2 GB at the same dimensionality and precision.
HierarchicalTokenPooling, in Sentence Transformers, groups similar token embeddings with Ward hierarchical clustering. It repeatedly merges the two groups that add the least within-group spread, then represents each remaining group by its mean and renormalizes. With pool_factor=4, documents keep about one quarter of their original vectors, so longer documents keep more capacity. By default the first token is not merged, and pooling is applied to document embeddings only. Queries are left unpooled. The library also supports a fixed cap on vectors per document and a distance threshold that can be calibrated to a target average.
The post is explicit about lineage. Pooling-aware training was studied in Stefan Josef's "Learn to Pool" and in LightOn's pooling regularization work. This article is a reproducible walk-through in Sentence Transformers, with published weights, not a claim to have invented the idea.
[1][2]A controlled comparison, with one important limit
Two models were trained from answerdotai/ModernBERT-base, each with a new 128-dimensional token projection. One, tomaarsen/mve-ModernBERT-base-token-pooling-baseline, used only a cached contrastive loss. The other, tomaarsen/mve-ModernBERT-base-token-pooling, wrapped that loss with unpooled and factor-4 pooling-aware objectives. Both ran for one epoch on the same 120,000 MS MARCO triplets, with seed 42 and matching initialization and optimization. The training token cap was 300. Short queries were expanded to at least 32 token positions. The author did not run a hyperparameter search. Only the loss differed.
Evaluation was on NanoBEIR@13, reported as mean nDCG@10 in percent, a standard score for how high the right documents appear in a ranked list. At pool_factor=4, the ordinary model scored 49.03 and the pooling-aware model scored 51.23, a gain of 2.20 points, while storing 74.8% fewer document vectors than unpooled retrieval. Retention, defined as pooled mean nDCG@10 divided by each model's own unpooled mean, rose from 94.6% to 97.8%. Unpooled retrieval also improved slightly, from 51.84 to 52.39.
The table shows the pattern at other budgets. At pool_factor=2, the gain was 1.64 points. At pool_factor=6, it was 2.78. With a hard cap of 16 vectors, the gain was 2.65 points. With a cap of 4, it was 5.78 points, from 37.22 to 43.00. The advantage was smaller, though still positive, when a distance threshold was calibrated to match average vector counts of 64, 32, or 16. The post notes that short documents pull those averages below the nominal caps.
The honest limit is the test bed. NanoBEIR is a smaller proxy for BEIR. The run is one epoch, one seed, one base model, and no hyperparameter search. That is a virtue for a controlled comparison, and a reason not to treat 97.8% as a universal constant.
[1][2]What to compress first
The practical advice in the post is an order of operations, not a slogan. If index size is the main constraint, start with embedding quantization to 4, 2, or 1 bits, for example with Fast-PLAID, which reduces storage per coordinate. Then apply token pooling, optionally with pooling-aware fine-tuning. Then consider Matryoshka token embeddings, which train the vectors to remain useful after truncation to a smaller width. These can be combined. MultiVectorMatryoshkaLoss can be wrapped with MultiVectorTokenPoolingLoss so the same encoder is trained for fewer dimensions and fewer document vectors.
A CUDA Ward clustering kernel is also mentioned, running 30 to 150 times faster than SciPy depending on the setting. That matters for indexing time. It does not change the quality claim.
[1][2]The fair objection
A fair reply is that anyone who already pools at inference knew there would be some quality loss, and that a 2.20-point gain at a four-times smaller vector budget, on a compact benchmark, is a useful engineering result rather than a new law of retrieval. The prior papers already argued that the encoder should be trained for the pooling it will face. This post mainly makes that recipe easy to run, and measures it cleanly.
That is a fair account of the novelty. It is also why the result is usable. If later runs on full BEIR, or on a private corpus, keep most of the retention gap, then the default should change. Do not train a multi-vector model as if every token vector will be stored, then pool only at index time and accept the loss as inevitable.
[1][2]What to watch in the next six months
The next checks are straightforward. Replicate the one-epoch ModernBERT comparison on full BEIR, and on at least one large private corpus, at the same pool_factor=4 budget. If retention stays close to 97.8% against a mid-90s baseline, pooling-aware training belongs in the default Sentence Transformers path for multi-vector indexes. If the gap shrinks, the blog will still have been useful as a careful measurement, not as a new default.
[1][2]