Multimodal RAG with ColPali: page images, multi-vector costs, and when they win

Multimodal RAG with ColPali changes the first stage of retrieval for documents whose meaning lives in layout: scanned forms, slide decks, tables and charts. Instead of parsing each page into text and chunks, you render the page as an image, run it through a vision-language model, and keep one embedding vector per image patch. A query is matched against those patch vectors with a late-interaction score called MaxSim. The pipeline loses its OCR, layout detection, captioning and chunking steps, and it gains a much larger index and a heavier query path.

The trade is real in both directions. On ViDoRe v1, the page-level benchmark that ColPali introduced, the model scores 81.3 nDCG@5 against 67.0 for the best parsed-text pipeline in the paper, and the gap is widest on figures, infographics and tables. The same design costs roughly 260 KB of float16 storage per page and forces brute-force scoring at query time. This article walks through the mechanism, the storage math, the compression options that make it practical, the vector stores that support it, and the cases where parsed text still wins. Figures are as of October 2026 unless a source is dated otherwise.

What does multimodal RAG with ColPali replace?

It replaces the parse pipeline for visually rich pages. A conventional PDF pipeline runs layout detection, OCR, figure captioning and then chunking before a text embedding model sees anything. Each step adds latency and errors, and each is a place where a table can lose its column headers. The ColPali approach skips all of that. The page image goes straight into a vision-language model, and the output is a set of vectors that stay tied to image regions.

The paper’s indexing numbers show the scale of the change. On one GCP g2-standard-8 machine with an NVIDIA L4, the parse path (layout 0.81 s, OCR 2.67 s, captioning 3.71 s, encoding 0.03 s) takes 7.22 s per page. ColPali takes 0.39 s per page and the SigLIP baseline 0.12 s. Skipping parsing is the single largest cost saving, and it also removes the OCR errors that would otherwise be baked into the index (ColPali paper).

The trade is not free. The same page now produces about a thousand vectors instead of one or a few chunks, and the query side needs a vision-language encoder too. In the paper, query encoding takes about 30 ms at batch size one, against about 2 ms for BGE-M3, while late-interaction scoring adds roughly 1 ms per 1,000 pages. Indexing gets cheaper and queries get more expensive, and storage cost sits between the two.

How does late-interaction MaxSim scoring work?

MaxSim scores a query against a page by letting every query token find its best match among the page’s vectors, then summing those best matches. For each query token, take the maximum dot product over all page vectors, and add the maxima together. Pages are encoded once, offline. At query time only the query is encoded, and matching runs against the stored patch vectors. That is why the design is called late interaction: query and page meet only at scoring time, not inside the encoder.

In ColPali, PaliGemma-3B with a SigLIP-So400m vision encoder splits each page into a 32 × 32 grid of 1,024 patches. Six instruction tokens are appended, and each of the 1,030 vectors is projected to 128 dimensions. The Vespa write-up describes the same layout. The core of the scoring is short:

import numpy as np

def maxsim(query_vecs, page_vecs):
    # query_vecs: (query_tokens, 128), page_vecs: (patches, 128)
    sims = query_vecs @ page_vecs.T      # (query_tokens, patches)
    return sims.max(axis=1).sum()        # best patch per query token, summed

Because the score is a sum over query tokens of per-token maxima, it can also be inspected. You can see which patch won each query token and draw a heatmap over the page. That helps when debugging retrieval, and it shows a reader why a page was returned, which matters when the answer has to be cited.

Why does multi-vector storage get so large?

A single-vector index stores one vector per document, so its size grows with the number of documents. A multi-vector index stores one vector per token or patch, so its size grows with documents times vectors per document. The ColPali paper reports 257.5 KB per page in float16, against 8.60 KB for a BGE-M3 chunk embedding. That is roughly 30 times the text figure, and the paper itself points to pooling and binarization as the remedy.

Scale that to a million pages and the table shows the spread. The figures come from the storage analysis in the EVIE paper (arXiv 2610.11553), which compares current ColQwen-family checkpoints with a single 2048-dimension baseline.

Two things follow. First, the vector count per page, not the dimension alone, drives cost, so a model with 2,304 vectors of 4,096 dimensions sits in a different storage class from one with 768 vectors of 320 dimensions. Second, HNSW graphs handle MaxSim poorly. The Qdrant docs note that HNSW is usually disabled for multivectors, so queries score candidates directly instead of walking a graph. That is workable for a few hundred candidate vectors per query and expensive for a full scan of a large corpus.

How do you make multi-vector retrieval affordable?

Three levers do most of the work: fewer vectors per page, fewer bits per vector, and a cheap first stage that narrows the candidate set before exact MaxSim runs. In practice teams combine them.

• Token pooling. Averaging neighbouring patch vectors shrinks the count. ColPali’s own measurement shows that pool factor 3 removes 66.7% of vectors and keeps 97.8% of performance. Sentence Transformers v6 ships hierarchical token pooling as a first-class option; on its 4,874-passage NQ example, a pool factor of 3 cuts 608,414 vectors to 204,407 (311.5 MB to 104.7 MB, float32).
• Quantization. Qdrant’s example of a million-document index with 1,024 × 128 vectors needs 512 GB at float32, and 16 GB at 1-bit, 21.3 GB at 1.5-bit and 32 GB at 2-bit, before rescoring. Qdrant’s turbo4 datatype stores vectors at 4 bits per dimension on disk; on five BEIR datasets with ColBERT multivectors it keeps 99.6% of float32 NDCG@20 through the same MaxSim rescoring path. Stacking 1-bit quantization on top of turbo4 cost more quality (recall@20 down 4.1%) and added 27.5% median latency.
• Binary with Hamming distance. Vespa reports that binary vectors with Hamming distance save 32 times the storage and run about 3.5 times faster than float dot products. On DocVQA nDCG@5, the float baseline scores 52.4, binary-only 49.5, and binary with a float rerank 51.6.
• PLAID. ColBERTv2’s residual compression engine, PLAID, cuts the late-interaction space footprint six to tenfold (ColBERTv2, arXiv 2112.01488).
• MUVERA. The NeurIPS 2024 paper converts each multi-vector set into a fixed-dimensional encoding, so a single-vector MIPS index can fetch candidates, followed by an exact MaxSim rerank. It reports about 10% higher recall and about 90% lower latency than PLAID.

Two-stage retrieval is the shape that works. A 2024 Qdrant experiment on 20,000 pages pooled each page’s 1,030 vectors to 38, retrieved the top 200, and reranked those with full-resolution MaxSim. The post reports a 13x retrieval speedup. Treat that as one vendor’s 2024 benchmark rather than a universal constant. The structure is the point: the cheap stage decides recall, and the exact stage decides order.

Which tools serve these models in 2026?

The paper’s own library, colpali-engine, is deprecated, and new projects are pointed to Sentence Transformers v6. Its MultiVectorEncoder loads ColPali, ColQwen2, PyLate and ColBERTv2 checkpoints and computes MaxSim with maxsim or meanmaxsim. Sentence Transformers v6.0.0 requires transformers v5 and torch 2.2 or later. For serving at scale, vLLM has a ColQwen3 implementation with token_embed pooling and MaxSim scoring, so the encoder can run behind the same inference stack as the rest of the RAG system. Check licences before you ship: ColPali is under Gemma terms, the ColQwen2 and ColQwen2.5 checkpoints and colSmol are Apache-2.0, and the TomoroAI colqwen3 models are Apache-2.0.

When do page images beat parsed text?

Page images win when the answer depends on layout or on non-text content: a figure, a chart legend, a table with merged headers, a scanned form. On ViDoRe v1 the ColPali paper reports 79.1 against 35.7 on ArxivQA, 81.8 against 71.9 on InfoVQA and 83.9 against 69.1 on TabFQuAD (nDCG@5). That benchmark deliberately tests figures and tables, so the gaps reflect what it measures.

The benchmarks have moved on. ViDoRe v1 is close to saturated, with top models above 90 nDCG@5. ViDoRe v2 adds blind contextual queries, long and cross-document queries, and four multilingual datasets. ViDoRe v3 (ACL 2026) is much harder: 10 datasets, about 26,000 pages and 3,099 human-verified queries in six languages. Visual retrievers beat textual ones and late interaction improves ranking, but current models still fail on non-textual elements and fine-grained visual grounding. The best reported score is about 66.75 nDCG@10 (EVIE-8B), against 61.60 for tomoro-colqwen3-embed-8b. Those are paper numbers. Check the model cards and the benchmark pages before quoting a leaderboard, because live views change.

Stay-with-text cases are real. Born-digital PDFs with clean text layers rarely need a vision model to find the right paragraph, and exact-match or field-level filters are easier to reason about over text. Parsed chunks also give you a span to cite. A few thousand pages on a CPU-only box is a poor fit for a multi-vector index. Scripts that stress OCR are a case to test, not a measured win. No ViDoRe split is a CJK corpus, so test those on your own pages.

Which vector stores support multivectors?

Support is uneven. Qdrant has a native multivector field with a max_sim comparator since v1.10.0, and it enforces a limit where sub-vectors times dimensions stay below 1,048,576, which a 128-dimension ColPali page meets easily. Vespa expresses MaxSim as a tensor ranking expression and supports binary vectors with Hamming distance. In Postgres, pgvector has no native multivector type; its feature request, issue 970, documents the workarounds of one row per token and application-side MaxSim. VectorChord provides the production route, with vector arrays, an @# MaxSim operator and a vchordrq index (VectorChord docs).

The table reflects only what the linked sources document. Check the current release of each product before you commit to a feature.

How do you decide, and what do you test first?

Start with your own pages, not a leaderboard. Build a small evaluation set of 100 to 300 real questions with page-level labels, run parsed text and page images through the same retrieval harness, and compare recall at the cut-off your generator actually uses. Then check these items:

1. Corpus size and growth: multiply pages by vectors per page and bytes per vector before you pick a store.
2. Page mix: how many pages depend on tables, figures or scans, and how many are clean born-digital text.
3. GPU budget: in the paper’s setup, query encoding costs roughly 15 times the text path, so plan for it on the query path, not only at indexing.
4. Reranking: decide whether a candidate stage with pooled or binary vectors will feed an exact MaxSim rerank.
5. Licences: confirm the checkpoint’s licence allows your use.
6. Citations: confirm the answer can point to a page region, and return the page image alongside the answer.

If I built a demo of this, I would start with a search box over rendered pages on my own Nuxt site, where I already run LLM features. That is a plan, not a running system. For the text baseline, read parsing PDFs for RAG and embedding quantization with binary and Matryoshka. For the store side, Qdrant covers the single-vector setup that remains the default. My AI engineering work is summarized on the AI engineer page.

Sources

1. ColPali: Efficient Document Retrieval with Vision Language Models (arXiv:2407.01449)
2. ViDoRe V2 benchmark (arXiv:2505.17166)
3. ViDoRe V3 benchmark (arXiv:2601.08620)
4. EVIE: storage and benchmark analysis for ColQwen-family models (arXiv:2610.11553)
5. Vespa: scaling ColPali to billions
6. Qdrant documentation: vectors, multivectors and the max_sim comparator
7. Qdrant: ColPali optimization with pooling and two-stage retrieval (2024)
8. Qdrant: turbo4 multivector search tutorial
9. Sentence Transformers v6.0.0 release notes
10. illuin-tech/colpali repository and model table
11. MUVERA: multi-vector retrieval via fixed dimensional encodings (NeurIPS 2024)
12. VectorChord documentation: indexing with MaxSim operators