RAG in 2026: hybrid retrieval, agentic search, or just a 1M-token context?

RAG in 2026 is no longer one architecture. Retrieval-augmented generation now means choosing between three ways of giving a model your knowledge: put all of it in a context window that holds a million tokens, retrieve the right chunks with a hybrid search pipeline, or let an agent search iteratively with tools. Each wins in a different situation, and the wrong choice costs either accuracy or money on every request.

This is the strategy piece: when each approach fits, what it costs per request, where it fails, and how to evaluate the choice. For the build itself (parsing, chunking, BM25, Reciprocal Rank Fusion, reranking, citations), see a production RAG pipeline, step by step.

What are the three ways to give an LLM your knowledge?

The three options are long context (send everything), retrieval (send the top chunks from an index), and agentic search (let the model call search tools until it has enough). They differ in who decides what the model reads: nobody, a ranking function, or the model itself.

• Long context. The whole knowledge base goes into the prompt, usually behind a prompt cache. No index, no chunking, no retrieval misses. As of September 2026, current Claude models (Opus 5.5, Sonnet 5, Fable 5.1) have a 1M-token context window.
• Hybrid retrieval. Documents are chunked and indexed ahead of time; each question runs BM25 and vector search, fuses and reranks the results, and sends the top chunks. One retrieval round per question.
• Agentic search. The model gets tools such as grep, file reads, a search API or a SQL endpoint, and decides what to look up next based on what it found. Several rounds per question.

When does a 1M-token context beat RAG?

A long context beats retrieval when the whole knowledge base fits comfortably and changes rarely. Anthropic's own guidance is that below 200,000 tokens, about 500 pages, you can "just include the entire knowledge base" in the prompt.

The price argument has changed. As of September 2026, Claude 4.6 and later models bill the full 1M-token window at standard rates; the pricing page puts it as "a 900k-token request is billed at the same per-token rate as a 9k-token request". Some arithmetic from the list prices: a 200,000-token prefix on Claude Opus 5.5 ($4 per million input tokens) costs $0.80 per uncached request, and $0.04 when it is read from the prompt cache at $0.20 per million. On Sonnet 5 ($2 input, $0.20 cache reads) the cached read is also $0.04. At that price, skipping the retrieval stack is a serious option.

The accuracy argument has not fully caught up. Chroma's Context Rot study (July 2025, 18 models) found that "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows", and that it degrades faster when the question and the answer share few words. The older "Lost in the Middle" result points the same way: information in the middle of a long context is used worse than information at the start or end. So a long context removes retrieval misses but adds distraction. It works best when the questions are broad ("summarize our refund policy across these documents") and worst for needle-like lookups in large, repetitive corpora.

Why is hybrid retrieval still the default for large corpora?

Once a corpus is far beyond what fits in context, or questions need precise lookups, retrieval is still the most reliable and cheapest option. The strongest published recipe combines contextual chunks, BM25 plus embeddings, and a reranker.

Anthropic's Contextual Retrieval post measured the effect of each layer on the top-20 retrieval failure rate. Prepending a short model-written context to each chunk before embedding ("contextual embeddings") took it from 5.7% to 3.7%. Adding contextual BM25 brought it to 2.9%. Adding a reranker brought it to 1.9%, a 67% reduction overall. The one-time preprocessing cost was about $1.02 per million document tokens with prompt caching.

Two things make this the default. First, cost per question is small and flat: one embedding call, two index lookups, one rerank, and a prompt of a few thousand tokens. Second, the retrieval step is inspectable. You can log which chunks were returned, compute recall against labelled questions, and enforce permissions inside the query. Neither is true of a million-token prompt.

What is agentic search, and when does it beat vectors?

Agentic search gives the model search tools and lets it iterate: query, read, refine, query again. It beats a vector index when the corpus changes constantly, has strong structure (paths, identifiers, schemas), and when the question needs several hops.

Coding agents are the clearest case. A community write-up of the agentic search pattern quotes Anthropic's Cat Wu on Claude Code: "We did use vector embeddings initially. They're really tricky to maintain because you have to continuously re-index… Claude is really good at agentic search." A codebase changes with every commit, identifiers are exact strings that grep finds reliably, and the agent can follow an import from one file to the next. An index would always be slightly stale, including for uncommitted changes.

The same write-up lists the costs honestly: more tokens across iterations, higher latency for complex questions, weaker semantic matching ("authentication" versus "login"), and a need for capable models. Agentic search also moves the stopping decision to the model, so it needs a budget; the mechanics are in the agent loop, explained. A practical middle ground is to give the agent a hybrid search tool as one of its tools. It then gets semantic recall when it needs it and exact lookups otherwise.

How do cost and latency compare across the three approaches?

Retrieval has the lowest and most predictable cost per question; long context is cheap only while the cache is warm; agentic search costs the most and varies the most. The table is qualitative except where a price is quoted.

Prompt caching is what makes the long-context column viable, and its details (5-minute versus 1-hour cache, what invalidates it) decide the real bill. Those are covered in cutting LLM cost and latency with caching, routing and batching.

Trade-offs: where each approach fails

Each option has a failure mode that the others avoid. Choosing well means knowing which failure your users would notice first.

• Long context fails on scale, permissions and precision. It stops working when the corpus grows past the window, it cannot serve users with different access rights from one cached prompt, and needle-like questions suffer from context rot.
• Hybrid retrieval fails silently. If the right chunk is not in the top k, the model answers from the wrong ones and sounds just as confident. Only a retrieval metric catches this.
• Agentic search fails on cost and stopping. Tokens grow with each round, and the model can stop after the first plausible hit or keep searching after it has the answer.
• None of them handles aggregation. "How many orders shipped late last quarter" is a SQL query. Give the model a query tool instead of documents.

How do you evaluate the choice?

Evaluate retrieval separately from generation, with the same labelled question set for every option. If you only grade final answers, you cannot tell a retrieval miss from a generation error.

Collect a few dozen real questions, label the passages that answer each one, and include questions with no answer. For retrieval, measure recall@k (Anthropic's failure rate is one minus recall@20). For long context, where there is no retrieval step, check whether the answer cites the right passage. For agentic search, log the tool calls and measure whether the right file or row was read at all, plus the number of rounds and tokens. Then grade the answers with a validated grader. The method for building and validating graders is in evals for LLM product features.

A decision checklist

1. Count your corpus in tokens with the provider's tokenizer, not in pages or words.
2. Under ~200K tokens and stable: try long context with prompt caching first and measure answer quality.
3. Different users see different documents: use retrieval with permission filters inside the query.
4. Large or growing corpus: hybrid retrieval with contextual chunks, BM25 plus vectors, and a reranker.
5. Code or structured files that change constantly: agentic search with grep, reads and a turn budget.
6. Counting and aggregation questions: give the model a SQL or API tool, not documents.
7. Log what the model read for every answer: chunk ids, files or cache prefix version.
8. Evaluate retrieval and answers separately on one labelled question set before and after every change.

If you're choosing between these for a product and want to talk it through, see AI engineering.

Sources

1. Anthropic: Introducing Contextual Retrieval (2024)
2. Chroma: Context Rot – How Increasing Input Tokens Impacts LLM Performance (2025)
3. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)
4. Claude docs: Pricing (long context, prompt caching, tokenizer)
5. Claude docs: Models overview
6. Awesome Agentic Patterns: Agentic search over vector embeddings