A production RAG pipeline, step by step: chunking, hybrid search and reranking

A RAG pipeline is the chain of steps that turns your documents into searchable chunks and, when a question arrives, finds the few passages a language model needs to answer with evidence. Retrieval-augmented generation (RAG) was introduced by Lewis et al. in 2020 as models that "combine pre-trained parametric and non-parametric memory": a generator plus a neural retriever over a dense vector index.

This article walks through a production pipeline one stage at a time: parsing, chunking, embeddings and an HNSW index, hybrid BM25 plus vector search merged with Reciprocal Rank Fusion (with working code), cross-encoder reranking, generation with citations, and the metrics that tell you which stage is failing. If you are still deciding whether you need retrieval at all, read RAG in 2026: hybrid retrieval, agentic search or long context first.

What is a RAG pipeline?

A RAG pipeline has two halves that share one index: an offline indexing path that parses, chunks, embeds and stores documents, and an online query path that retrieves, fuses, reranks and generates. Most quality problems start in the offline half, even though they only show up in the answers.

Each stage has one job, and each can be measured on its own. That separation matters when an answer is wrong, because the cause can sit in four different places: a chunk that was never created (parsing), a chunk that exists but was not retrieved (retrieval), a chunk that was retrieved but ranked below the cutoff (fusion or reranking), or a correct chunk the model ignored (generation). If you only measure the final answer, you cannot tell these apart.

How should you parse and chunk documents?

Parse documents into clean text with their structure and metadata intact, then split them along their own structure into chunks of a few hundred tokens that make sense on their own. Chunking is the cheapest stage to change and one of the most influential.

Parsing: keep structure and metadata

• PDFs lose reading order in multi-column layouts and repeat headers and footers on every page. Strip the repeats and check a sample of extracted pages by eye before you tune anything downstream.
• Tables break naive splitters. A cell that says "4.2" means nothing without its row and column headers, so either keep small tables whole as Markdown or write one line per row that repeats the column names.
• Metadata belongs on every chunk: document id, title, heading path, source URL, last-modified date, language, and the groups allowed to read it. Access rules stored at index time are what make permission filtering possible later.
• A content hash per document lets you re-index only what changed.

Chunking strategies compared

The best public comparison I know is Chroma's Evaluating Chunking Strategies for Retrieval (Smith and Troynikov, July 2024). It measured token-level recall, precision and IoU, and found that a recursive character splitter at 200 tokens with no overlap performed consistently well, while the then-default OpenAI Assistants setting of 800-token chunks with 400 tokens of overlap had slightly below-average recall and the lowest scores on the other metrics. Their conclusion is the useful part: "The choice of chunking strategy can have significant impact on retrieval performance." Treat the numbers as a starting point and measure on your own documents.

Anthropic's Contextual Retrieval is the strongest version of contextual headers: a model writes 50–100 tokens that situate each chunk in its document, and that text is prepended before both embedding and BM25 indexing. In their tests this cut the top-20 retrieval failure rate from 5.7% to 3.7%, and to 2.9% combined with BM25, at a one-time cost of about $1.02 per million document tokens with prompt caching. My default is cheaper: structure-aware splitting at a few hundred tokens with a deterministic header (title plus heading path), and model-written context only when retrieval evals show chunks failing because they lack it.

Which vector index should you use?

For most teams an HNSW index inside a database they already run is enough. pgvector adds HNSW and IVFFlat indexes to PostgreSQL, so vectors, full-text columns and access-control columns live in one table and one transaction.

HNSW (Malkov and Yashunin, 2016) builds a multi-layer proximity graph and searches it from coarse to fine layers, which scales roughly logarithmically. It is approximate: it trades a little recall for a lot of speed. In pgvector, HNSW has better query performance than IVFFlat but slower builds; the build parameters default to m = 16 and ef_construction = 64, and the query-time candidate list hnsw.ef_search defaults to 40.

-- One table for vectors, full-text and permissions (pgvector + Postgres FTS)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id           bigserial PRIMARY KEY,
  doc_id       text NOT NULL,
  allowed      text[] NOT NULL,          -- groups that may read this chunk
  heading_path text,
  content      text NOT NULL,
  embedding    vector(1024),             -- match your embedding model
  tsv          tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks USING gin (allowed);

-- Vector leg of the hybrid query, filtered by the user's groups
SET hnsw.iterative_scan = relaxed_order;
SELECT id FROM chunks
WHERE allowed && $1::text[]
ORDER BY embedding <=> $2
LIMIT 50;

The iterative_scan line matters for permissions. The pgvector README warns that with approximate indexes "filtering is applied after the index is scanned": if a condition matches 10% of rows, HNSW with the default ef_search of 40 returns only about 4 matching rows. Iterative index scans (added in 0.8.0) keep scanning until enough rows match. For the embedding model itself, Anthropic's tests found Voyage and Gemini embeddings performed best, but run your own comparison, and remember that switching models means re-embedding the entire corpus.

How does hybrid search with BM25 and vectors work?

Hybrid search runs a lexical BM25 query and a vector query in parallel and merges the two ranked lists. Reciprocal Rank Fusion (RRF) is the simplest robust merge, because it uses ranks and ignores the raw scores, which are not comparable.

Vectors match meaning, so they find paraphrases, but they are weak on exact tokens: error codes, SKUs, part numbers, names and version strings. BM25 matches exact terms, weights rare terms higher and saturates repeated ones. Elasticsearch uses BM25 as its default similarity, with k1 = 1.2 and b = 0.75. One caveat for Postgres users: the built-in ts_rank and ts_rank_cd functions are not BM25. The PostgreSQL documentation states that they "do not use any global information", so there is no inverse document frequency. That is fine to start with; if lexical quality matters, use a search engine or a BM25 extension for that leg.

RRF comes from Cormack, Clarke and Büttcher (SIGIR 2009). Each document scores the sum of 1 / (k + rank) over every list it appears in, with k = 60, a value the authors "fixed during a pilot investigation and not altered during subsequent validation". In their experiments RRF consistently beat any individual system and the standard Condorcet Fuse method.

// Reciprocal Rank Fusion (Cormack et al., 2009) – TypeScript
type Hit = { id: string }

export function reciprocalRankFusion(lists: Hit[][], k = 60, limit = 50) {
  const scores = new Map<string, number>()
  for (const list of lists) {
    list.forEach((hit, index) => {
      const rank = index + 1 // ranks start at 1
      scores.set(hit.id, (scores.get(hit.id) ?? 0) + 1 / (k + rank))
    })
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .slice(0, limit)
    .map(([id, score]) => ({ id, score }))
}

// const fused = reciprocalRankFusion([bm25Top50, vectorTop50])

Fetch more candidates from each leg than you plan to keep, for example 50 each, so that a document ranked 30th by one retriever and 5th by the other can still surface. pgvector's README points to the same two merge options this article uses: RRF, or a cross-encoder over the combined candidates.

How do reranking and cited generation work?

Rerank the fused candidates with a cross-encoder, pass only the best few chunks to the model, and require citations plus an explicit "the sources don't say" answer. Reranking buys precision; citations make the answer checkable.

Reranking with a cross-encoder

An embedding model is a bi-encoder: it encodes the query and each chunk separately. A cross-encoder reads the query and one chunk together and outputs a relevance score. The Sentence Transformers documentation sums up the trade-off: "Cross-Encoders achieve better performances than Bi-Encoders", but they produce no embeddings, so they cannot search a corpus. The standard answer is retrieve-then-rerank: fetch around 100 candidates cheaply, then score each pair with the cross-encoder. Anthropic's Contextual Retrieval tests added a reranker on top of contextual embeddings and BM25 and reduced the top-20 failure rate to 1.9%. Reranking adds one model inference per candidate, so cap the candidate count and watch the latency.

Generation with citations and an "I don't know"

Give each chunk an id and a title, keep the count small, and put the strongest chunks first. The "Lost in the Middle" study (Liu et al., 2023) found that performance "is often highest when relevant information occurs at the beginning or end" of the context and degrades in the middle. Tell the model to answer only from the sources and to say so when they don't contain the answer, then test that behavior with unanswerable questions.

If you use Claude, the Citations feature does the bookkeeping: enable citations on document blocks and the response points to the exact passages it used, and the returned cited_text does not count toward output tokens. Two details from the docs: to cite specific sentences from RAG chunks, put each chunk in its own plain-text document; and citations cannot be combined with structured outputs (the API returns a 400 error).

How do you evaluate a RAG pipeline?

Evaluate retrieval and generation separately. Retrieval gets ranking metrics against a labelled set of relevant chunks; generation gets faithfulness and correctness checks against the retrieved context and a reference answer.

Anthropic's "failure rate" is simply one minus recall@20. For generation, RAGAS defines faithfulness as "Number of claims in the response supported by the retrieved context / Total number of claims in the response", from 0 to 1. Build a golden set from real user questions, label the chunk ids that answer each one, and include questions with no answer in the corpus and questions the test user is not allowed to see. Rerun the retrieval metrics on every chunking, embedding or fusion change; they are cheap and deterministic. Model-graded metrics need validating against human labels, which is covered in evals for LLM product features.

Operating the pipeline, and when not to build one

A RAG pipeline is a data system: it needs incremental re-indexing, deletions, permission filters and versioned indexes. For a small corpus it may not be worth building at all.

• Re-index incrementally. Compare content hashes, re-chunk only changed documents, and delete chunks of removed documents. Stale chunks of deleted pages are a common source of confidently wrong answers.
• Version the index. A new embedding model or chunking strategy means a full rebuild. Build the new index next to the old one, run the retrieval evals on both, then switch.
• Filter permissions inside retrieval. Apply the user's groups in both the BM25 and the vector query. Filtering after generation is too late: the model has already read the text.
• Track freshness. Store last-modified dates, show them with citations, and alert on sources that stopped syncing.

When not to build one: Anthropic's advice is that for a knowledge base under 200,000 tokens (about 500 pages) you can "just include the entire knowledge base" in the prompt, with prompt caching keeping the cost down. For codebases, agents searching with grep and file reads often do well without an index. And questions that aggregate over everything ("how many contracts expire this year") are database queries, not top-k retrieval. The trade-offs between these options are the subject of the strategy article on RAG in 2026, and the cost side of long prompts is in prompt caching, routing and batching.

RAG pipeline checklist

1. Inspect parsed output by eye for a sample of PDFs and tables before tuning anything else.
2. Store metadata on every chunk: document id, heading path, source URL, last-modified date, allowed groups.
3. Start with structure-aware chunks of a few hundred tokens plus a title and heading-path header.
4. Run BM25 and vector search in parallel, about 50 candidates each, and merge them with RRF (k = 60).
5. Rerank with a cross-encoder and pass only the best few chunks, strongest first.
6. Require citations and an explicit answer for "not in the sources", and test both.
7. Measure recall@k and MRR on a labelled golden set for every retrieval change; check faithfulness separately.
8. Enforce permissions inside the retrieval query, with iterative index scans if you filter an HNSW index.
9. Version and rebuild the index side by side when the embedding model or chunking changes.

If you're building retrieval into a product and want a second pair of eyes on the pipeline or its evals, see AI engineering.

Sources

1. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
2. Cormack, Clarke and Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009)
3. Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using HNSW graphs
4. pgvector README
5. Elasticsearch: similarity settings (BM25 default)
6. PostgreSQL: controlling text search (ranking)
7. Chroma: Evaluating Chunking Strategies for Retrieval (2024)
8. Anthropic: Introducing Contextual Retrieval (2024)
9. Sentence Transformers: Cross-Encoders
10. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)
11. Claude docs: Citations
12. RAGAS: Faithfulness metric