Contextual retrieval and late chunking: giving RAG chunks back their context

Chunking is a lossy step. A chunk that says "the maximum load is 40 kg" was perfectly clear inside its manual, where the heading two pages earlier named the product. Cut out and embedded alone, it no longer mentions the product, the variant or the document it came from, and a query that names the product may never find it.

Three families of fixes appeared since 2024: contextual retrieval, which rewrites each chunk with an LLM; late chunking, which embeds the whole document before splitting; and contextualized embedding models sold as an API. This post compares what each changes, what the published numbers say, what they cost, and the cheap baseline I try first. It builds on the chunking and hybrid search pipeline described earlier.

Why do chunks lose their context?

Embedding models see only the text you pass them. When a document is split first and each piece is embedded separately, references that point outside the chunk, such as "it", "this variant", "the above table" or an implicit product name, carry no meaning. BM25 suffers the same way: the keyword that would match is in the heading, not in the chunk.

Bigger chunks are the obvious workaround, but they dilute the vector and send more irrelevant text to the model. Smarter boundaries help less than one would hope: a 2024 study by Qu, Tu and Bao concluded that the extra cost of semantic chunking is not justified by consistent gains over fixed-size chunks. The fixes below leave boundaries alone and add context instead.

How does contextual retrieval work?

Anthropic published contextual retrieval on 19 September 2024. For every chunk, a small model receives the full document and the chunk and is asked for a short context that situates the chunk for search, answering with the context and nothing else. The result, usually 50 to 100 tokens, is prepended to the chunk. The combined text is then embedded and also indexed for BM25, which Anthropic calls contextual embeddings and contextual BM25.

The reported effect is large. Anthropic measured the top-20 retrieval failure rate, one minus recall@20, averaged over codebases, fiction, arXiv papers and science papers:

Two details from the same post are easy to miss. Generic document summaries prepended to chunks gave very limited gains, so the context must be chunk-specific. And the reranked setup retrieved about 150 candidates, reranked them and kept the top 20, which performed better than passing 5 or 10 chunks.

import anthropic

client = anthropic.Anthropic()

def chunk_context(document: str, chunk: str) -> str:
    """One call per chunk; the document block is cached and reused for every chunk."""
    msg = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=150,
        messages=[{"role": "user", "content": [
            {"type": "text", "text": f"<document>\n{document}\n</document>",
             "cache_control": {"type": "ephemeral"}},
            {"type": "text", "text": (
                f"Here is a chunk from the document:\n<chunk>\n{chunk}\n</chunk>\n"
                "Write a short context that situates this chunk within the document "
                "to improve search retrieval. Answer only with the context.")},
        ]}],
    )
    return msg.content[0].text.strip()

# Index context and chunk together, but keep both fields for display and re-runs.
record = {"context": ctx, "chunk": chunk, "index_text": f"{ctx}\n\n{chunk}"}

What does contextualization cost?

Anthropic estimated about $1.02 per million document tokens, one-off, assuming 8,000-token documents, 800-token chunks, 50 tokens of instructions and 100 tokens of context per chunk, with the document cached. That figure used Claude 3 Haiku prices from 2024. I recalculated it with the same assumptions and the current list prices from Anthropic's pricing page: per million document tokens that is 125 documents and 1,250 chunks, one cache write per document, nine cache reads, the chunk and instructions as normal input, and the context as output.

Either way it is an indexing cost, paid again only when a document changes, and small next to the engineering time. The catch is change: when one paragraph of a document changes, the context of every chunk in it may change too, so re-contextualize the whole document, not only the edited chunk.

How does late chunking work?

Late chunking, proposed by Günther and colleagues at Jina AI in September 2024, moves the split to the end. The long-context embedding model reads the whole document, or as much as fits, and produces one vector per token. Each token vector already reflects the surrounding text through attention. Only then are the chunk boundaries applied: the token vectors inside each chunk are mean-pooled into one chunk embedding. The paper's phrase is chunking applied after the transformer and just before mean pooling.

Jina's original post shows the effect with a Wikipedia text about Berlin: the sentence about the population scored 0.708 cosine similarity to the query "Berlin" with naive chunking and 0.825 with late chunking, because "its" now carried the city. The paper's benchmarks are more modest. Averaged over three models and four BeIR datasets, late chunking improved nDCG@10 by about 1.8 points with fixed-size chunks, 1.9 with sentence chunks and 1.5 with semantic chunks. The biggest single jump was NFCorpus, from 23.5 to 30.0 with jina-embeddings-v2-small.

The paper is also honest about limits. Gains are largest with small chunks, and on some reading-comprehension sets naive chunking won with large chunks. On synthetic needle and passkey tasks, where the surrounding text is unrelated filler, late chunking did not help. For documents longer than the model's window, the authors split them into overlapping macro chunks and late-chunk each one.

import torch
from transformers import AutoModel, AutoTokenizer

# Any mean-pooling, long-context embedding model whose token outputs you can read.
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModel.from_pretrained(MODEL)

def late_chunk(text: str, spans: list[tuple[int, int]]) -> list[torch.Tensor]:
    """spans are character ranges from your normal chunker."""
    enc = tok(text, return_offsets_mapping=True, truncation=True,
              max_length=8192, return_tensors="pt")
    offsets = enc.pop("offset_mapping")[0].tolist()
    with torch.no_grad():
        tokens = model(**enc).last_hidden_state[0]      # one vector per token
    out = []
    for start, end in spans:
        idx = [i for i, (s, e) in enumerate(offsets) if e > s and s >= start and e <= end]
        v = tokens[idx].mean(dim=0)                     # pool only this chunk's tokens
        out.append(v / v.norm())
    return out

What about contextualized embedding APIs?

The third option packages the idea as a model. Voyage released voyage-context-3 in July 2025 and voyage-context-4 in June 2026: you send all chunks of a document together and get one vector per chunk, each computed with the whole document in view. The API takes a list of documents, each a list of chunks, with limits of 1,000 inputs, 120,000 total tokens and 16,000 chunks per request. Queries go through the same endpoint as single-element lists. voyage-context-4 costs $0.12 per million tokens, can chunk for you, and handles documents beyond 32,000 tokens by splitting them.

import voyageai

vo = voyageai.Client()

docs = [["Chunk 1 of manual A ...", "Chunk 2 of manual A ..."],
        ["Chunk 1 of manual B ..."]]
res = vo.contextualized_embed(inputs=docs, model="voyage-context-4",
                              input_type="document")
chunk_vectors = [r.embeddings for r in res.results]   # one list per document

q = vo.contextualized_embed(inputs=[["maximum load of the X200 shelf"]],
                            model="voyage-context-4", input_type="query")
query_vector = q.results[0].embeddings[0]

Voyage reports that voyage-context-3 beat Anthropic-style contextual retrieval by 6.76% and Jina late chunking by 23.66% on chunk-level retrieval. Treat those as vendor numbers, and note that the voyage-context-3 post itself lists two different sets of percentages in its summary and results sections. The architectural point is solid either way: you get document context without an LLM call per chunk, at the price of a vendor lock on both indexing and queries.

How do the three approaches compare?

When is added context not worth it?

• Self-contained records. Product data, FAQ entries and tickets already name their subject. Adding context repeats what is there.
• Small knowledge bases. Anthropic notes that below about 200,000 tokens, roughly 500 pages, putting everything in the prompt can be simpler than retrieval. I weighed that trade-off in RAG in 2026.
• Unrelated surroundings. Collections of independent notes in one file give the model nothing useful to propagate, as the needle results in the late chunking paper show.
• Fast-changing documents. If documents change daily, contextual retrieval means re-running the LLM over whole documents daily. Late chunking or a contextualized API costs only an embedding pass.

What should the implementation get right?

1. Store context and chunk separately. Index the combined text, but keep both fields. Anthropic notes that generation may improve if the model can tell the added context apart from the original text, and you need the raw chunk for citations.
2. Version the context prompt. The prompt is now part of your index. Changing it means re-contextualizing, so give it a version and store that version with each chunk.
3. Customize for your domain. Anthropic suggests domain-specific prompts, for example with a glossary, can beat the generic one. For a B2B catalogue I would tell the model to always name the product family and variant.
4. Measure retrieval, not vibes. Compare recall@20 and nDCG@10 on labelled queries before and after, as in Evaluating RAG. The published gains are averages over other people's data.

Which one should you pick?

My rule of thumb: if you run hybrid search, contextual retrieval is the default, because it is the only option that also fixes keyword matches and it works with any embedding model. If your corpus is large and changes often, or documents must not go to an LLM provider, late chunking with a self-hosted long-context model gives part of the benefit at embedding cost. If you are already on a vendor that offers contextualized embeddings and accept the lock-in, that is the least engineering. In all three cases, start from the title and heading prefix and keep the reranker, which contributed the last step from 49% to 67% in Anthropic's numbers.

Sources

1. Anthropic: Introducing Contextual Retrieval (Sep 2024)
2. Günther et al.: Late Chunking, Contextual Chunk Embeddings Using Long-Context Embedding Models (2024, rev. 2025)
3. Jina AI: Late Chunking in Long-Context Embedding Models (Aug 2024)
4. Voyage AI: Introducing voyage-context-3 (Jul 2025)
5. Voyage AI: voyage-context-4 (Jun 2026)
6. Voyage AI: Contextualized chunk embeddings documentation
7. Anthropic: Claude API pricing
8. Qu, Tu and Bao: Is Semantic Chunking Worth the Computational Cost? (2024)