[{"data":1,"prerenderedAt":747},["ShallowReactive",2],{"blog-code-rag-repository-indexing-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":25,"sources":37,"cover":74,"og":75,"expertise":76,"locales":77,"lang":78,"title":81,"description":82,"coverAlt":83,"metaTitle":84,"takeaways":85,"faq":91,"toc":107,"blocks":132,"others":497},"code-rag-repository-indexing","2026-10-01",8,"rag",[9,10,11,12,13],"RAG","Code search","Tree-sitter","Embeddings","Coding agents",[15,16,17,18,19,20,21,22,23,24],"RAG over code","AST chunking","tree-sitter code chunking","repository map","code embeddings","semantic code search","codebase indexing","repo map tokens","incremental code index","CoIR benchmark",[26,29,32,34],{"name":27,"url":28},"Retrieval-augmented generation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRetrieval-augmented_generation",{"name":30,"url":31},"Abstract syntax tree","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FAbstract_syntax_tree",{"name":11,"url":33},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTree-sitter_%28parser_generator%29",{"name":35,"url":36},"Merkle tree","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMerkle_tree",[38,41,44,47,50,53,56,59,62,65,68,71],{"title":39,"url":40},"cAST: AST-based chunking for code RAG (arXiv 2506.15655)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.15655",{"title":42,"url":43},"Aider docs: repository map","https:\u002F\u002Faider.chat\u002Fdocs\u002Frepomap.html",{"title":45,"url":46},"Aider blog: building a better repository map with tree-sitter (2023)","https:\u002F\u002Faider.chat\u002F2023\u002F10\u002F22\u002Frepomap.html",{"title":48,"url":49},"Anthropic Engineering: effective context engineering for AI agents (Sep 2025)","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Feffective-context-engineering-for-ai-agents",{"title":51,"url":52},"Cursor blog: securely indexing large codebases (Jan 2026)","https:\u002F\u002Fcursor.com\u002Fblog\u002Fsecure-codebase-indexing",{"title":54,"url":55},"turbopuffer: Cursor customer story","https:\u002F\u002Fturbopuffer.com\u002Fcustomers\u002Fcursor",{"title":57,"url":58},"Kilo Code docs: codebase indexing","https:\u002F\u002Fkilo.ai\u002Fdocs\u002Fcustomize\u002Fcontext\u002Fcodebase-indexing",{"title":60,"url":61},"Serena on GitHub","https:\u002F\u002Fgithub.com\u002Foraios\u002Fserena",{"title":63,"url":64},"Voyage AI docs: pricing","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fpricing",{"title":66,"url":67},"Qwen blog: Qwen3-Embedding (Jun 2025)","https:\u002F\u002Fqwenlm.github.io\u002Fblog\u002Fqwen3-embedding\u002F",{"title":69,"url":70},"CoIR: a benchmark for code information retrieval (arXiv 2407.02883)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2407.02883",{"title":72,"url":73},"CocoIndex blog: indexing a codebase","https:\u002F\u002Fcocoindex.io\u002Fblogs\u002Findex-codebase-v1","\u002Fimages\u002Fblog\u002Fcode-rag-repository-indexing\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fcode-rag-repository-indexing\u002Fog.jpg","ai-engineer",[78,79,80],"en","de","hu","RAG over code: AST chunking, repo maps and code embeddings that fit an agent","How coding agents find code: AST chunking with tree-sitter, repo maps, Merkle-tree index sync, code embedding models in 2026, and when grep beats an index.","A branching code tree feeds a row of index cards.","RAG over code: AST chunks and repo maps · Balázs Csorba",[86,87,88,89,90],"Line-based chunking splits functions and separates imports from the code that needs them; the cAST paper measured +4.3 Recall@5 on RepoEval and +2.67 Pass@1 on SWE-bench from AST-based chunking.","Tree-sitter is the common parser: chunk by function, class or method, merge small siblings, split oversized nodes, and store path, line range, symbol name and signature with each chunk.","A repo map needs no embeddings: Aider ranks tree-sitter symbols over a file dependency graph and fits signatures into a 1k-token default budget.","Index freshness is the hard part: Cursor syncs only changed branches of a Merkle tree, Kilo Code uses file watchers and hash caches, and CocoIndex re-embeds only changed files.","Grep and glob never go stale and cost tokens per search; an index costs storage, embedding spend and freshness work. Many teams need both: grep for exact names, an index for meaning.",[92,95,98,101,104],{"q":93,"a":94},"What is RAG over code?","Retrieval-augmented generation where the corpus is a code repository. Instead of splitting text into paragraphs, the system splits code along syntax, usually functions and classes, indexes those chunks with embeddings or symbol maps, and gives the model the most relevant pieces for a task.",{"q":96,"a":97},"Why use AST chunking instead of fixed line windows?","Fixed windows cut functions in half and separate decorators and imports from the code they belong to. Chunking along the abstract syntax tree keeps whole definitions together. The cAST paper reports +4.3 points Recall@5 on RepoEval and +2.67 points Pass@1 on SWE-bench from recursive AST chunking with sibling merging.",{"q":99,"a":100},"What is a repo map?","A compact summary of a repository’s classes and functions with their signatures, built with tree-sitter. Aider ranks symbols over a graph of file dependencies and includes only what fits a token budget, 1k tokens by default, so the model sees structure without file bodies.",{"q":102,"a":103},"Which embedding model should I use for code in 2026?","As of October 2026, code-tuned options include voyage-code-4 at $0.12 per million tokens and Qwen3-Embedding in 0.6B, 4B and 8B sizes for self-hosting; jina-embeddings-v4 reports 71.59 on CoIR under a research license. Evaluate two or three candidates on queries from your own repository before choosing.",{"q":105,"a":106},"Do coding agents need a vector index at all?","Not always. Claude Code navigates with grep and glob just in time, which avoids stale indexes and syntax-tree complexity but is slower than pre-computed retrieval. An index pays off in large or unfamiliar repositories and for questions phrased by meaning rather than by name.",[108,111,114,117,120,123,126,129],{"id":109,"title":110},"why-line-based-chunking-fails-on-code","Why line-based chunking fails on code",{"id":112,"title":113},"ast-chunking-with-tree-sitter","AST chunking with tree-sitter",{"id":115,"title":116},"repo-maps-symbol-context-without-embeddings","Repo maps: symbol context without embeddings",{"id":118,"title":119},"how-commercial-agents-index-code","How commercial agents index code",{"id":121,"title":122},"code-embedding-models-in-2026","Code embedding models in 2026",{"id":124,"title":125},"measuring-code-retrieval-and-keeping-it-fresh","Measuring code retrieval and keeping it fresh",{"id":127,"title":128},"when-agentic-grep-beats-an-index","When agentic grep beats an index, and a checklist",{"id":130,"title":131},"sources","Sources",[133,140,149,152,155,163,169,177,178,189,191,198,199,210,217,229,230,236,243,261,265,330,331,349,367,370,371,378,389,400,407,408,419,422,450,458,459],{"type":134,"content":135},"paragraph",[136,139],{"tag":137,"children":138},"strong",[15]," is retrieval-augmented generation where the corpus is a repository, and it fails in different ways than RAG over documents. Code has hard boundaries (functions, classes, imports), exact identifiers that matter more than paraphrase, and a call graph that connects files. Chunk it like prose and the model gets half a function and no imports; index it once and forget it, and the agent confidently reads code that was deleted yesterday.",{"type":134,"content":141},[142,143,148],"This article covers structural chunking with tree-sitter, repo maps that need no embeddings, how Cursor and Kilo Code keep an index fresh, the code embedding models available as of October 2026, how to measure retrieval on your own repository, and when grep beats any index. Deciding what goes into an agent’s context in the first place is covered in ",{"tag":144,"to":145,"children":146},"link","\u002Fblog\u002Fagents-md-skills-mcp-cli-decision-matrix",[147],"the AGENTS.md, skills, MCP and CLI decision matrix","; this one is about how the code gets found.",{"type":150,"level":151,"id":109,"text":110},"heading",2,{"type":134,"content":153},[154],"Line-based chunking fails because code boundaries do not fall on fixed line counts. A 40-line window splits a function in the middle, leaves a decorator in one chunk and its class in the next, and separates imports from the code that uses them. The retriever then ranks fragments that no longer mean anything on their own.",{"type":134,"content":156},[157,158,162],"The cAST paper measured the effect. It describes line-based heuristics as ones that “often break semantic structures, splitting functions or merging unrelated code,” and replaces them with recursive chunking along the abstract syntax tree, merging small sibling nodes until a size limit. That change improved Recall@5 by 4.3 points on RepoEval retrieval and Pass@1 by 2.67 points on SWE-bench generation (",{"tag":159,"href":40,"children":160},"a",[161],"cAST","). The gain is modest in absolute terms, but it comes from a parser, not from a bigger model.",{"type":164,"src":165,"alt":166,"wide":167,"caption":168},"image","\u002Fimages\u002Fblog\u002Fcode-rag-repository-indexing\u002Fdiagram-5.webp","Line windows versus AST chunks Two columns for the same source file. Column A, fixed line windows: window 1 holds the imports and the first half of parse(); window 2 holds the rest of parse() and a decorator cut off from its class; window 3 holds the class body without its decorator. Column B, AST nodes: a module chunk with imports and file-level constants; a function chunk with the whole parse() body plus path, line range and signature; a class chunk for Order with its decorator kept. A footer notes the cAST results: +4.3 Recall@5 on RepoEval and +2.67 Pass@1 on SWE-bench.",true,"Fixed line windows cut definitions apart; AST chunks keep each function and class whole and carry their location and signature.",{"type":134,"content":170},[171,172,176],"Code retrieval is also not text retrieval for a second reason: queries mix natural language with exact identifiers. A question like “where do we retry the ERP call” needs semantic matching, while “who calls ",{"tag":173,"children":174},"code",[175],"createSalesOrder","” needs an exact symbol lookup. A good setup serves both, which is why most of the systems below combine structure with search.",{"type":150,"level":151,"id":112,"text":113},{"type":134,"content":179},[180,181,184,185,188],"Tree-sitter is the common parser across these systems: it parses a file into a concrete syntax tree, and query files such as ",{"tag":173,"children":182},[183],"tags.scm"," define what counts as a definition or a reference for each language (",{"tag":159,"href":46,"children":186},[187],"Aider","). A chunker walks the top-level nodes, emits functions, classes and methods as chunks, merges small neighbors so a file of short helpers does not become fifty tiny chunks, and splits oversized nodes by recursing into their children.",{"type":173,"code":190},"# Sketch: AST chunks with sibling merging, the idea behind cAST\nfrom tree_sitter import Language, Parser\nimport tree_sitter_python as tspython\n\nparser = Parser(Language(tspython.language()))\nMAX_CHARS = 1000  # Kilo Code documents 100 to 1,000 characters per block\n\ndef chunks(src: bytes, path: str):\n    buf = []\n    for node in parser.parse(src).root_node.children:\n        text = src[node.start_byte:node.end_byte].decode()\n        if len(text) > MAX_CHARS and node.child_count:\n            yield from split_children(node, src, path)   # oversized: recurse\n        elif sum(len(t) for t, _ in buf) + len(text) \u003C= MAX_CHARS:\n            buf.append((text, node))                     # merge small siblings\n        else:\n            yield emit(buf, path)                        # path, lines, symbol, signature\n            buf = [(text, node)]\n    if buf:\n        yield emit(buf, path)",{"type":134,"content":192},[193,194,197],"Kilo Code, from the Cline and Roo lineage, documents concrete settings for this: tree-sitter parsing into semantic blocks of 100 to 1,000 characters, large functions split, and a line-based fallback for files the parser does not understand (",{"tag":159,"href":58,"children":195},[196],"Kilo Code","). Whatever the size limits, store metadata with every chunk: the file path, the line range, the symbol name and the signature. The agent needs the path and lines to open the real file, and the signature makes the chunk readable in a result list without its body.",{"type":150,"level":151,"id":115,"text":116},{"type":134,"content":200},[201,202,205,206,209],"A repo map gives the model the shape of a repository without any vector index. Aider builds one with tree-sitter: the key classes and functions with their types and call signatures. It selects what fits a token budget with a graph-ranking algorithm over a graph where each file is a node and dependencies are edges. The ",{"tag":173,"children":203},[204],"--map-tokens"," setting defaults to 1k tokens, and the map expands when no files have been added to the chat yet (",{"tag":159,"href":43,"children":207},[208],"Aider docs",").",{"type":134,"content":211},[212,213,216],"The map holds signatures, not bodies, which is why it is cheap. Aider’s design note from 2023 explains that tree-sitter replaced ctags because it shows full signatures straight from the source and ships language support as installable packages (",{"tag":159,"href":46,"children":214},[215],"Aider blog","). The retrieval pattern is marks first: the model reads the map, picks the files it needs, and only then reads them in full. For many tasks in a repository you already know, that is enough, and nothing ever has to be re-embedded.",{"type":134,"content":218},[219,220,223,224,228],"Language servers go one step further. Serena is an MCP toolkit for symbol-level retrieval and editing whose default backend is the Language Server Protocol, covering more than 40 languages (",{"tag":159,"href":61,"children":221},[222],"Serena","). Instead of searching text, the agent asks for a symbol’s definition or references and gets exact answers. How much that saves in practice is covered in ",{"tag":144,"to":225,"children":226},"\u002Fblog\u002Ftoken-saving-tools-coding-agents-top-20",[227],"the token-saving tools for coding agents",".",{"type":150,"level":151,"id":118,"text":119},{"type":134,"content":231},[232,233,209],"Cursor indexes with a Merkle tree. Each file gets a SHA-256 hash and each folder a hash of its children, so after an edit only the changed branches diverge and need to be synced. In a 50,000-file workspace, the filenames and hashes alone add up to roughly 3.2 MB. Changed files are split into syntactic chunks, embeddings are cached by chunk content, and a similarity hash finds an existing index to reuse, with content proofs so a client only gets results for files it can actually read (",{"tag":159,"href":52,"children":234},[235],"Cursor",{"type":134,"content":237},[238,239,242],"The vector layer behind it is turbopuffer, which describes Cursor’s retrieval as more than a trillion vectors with one namespace per codebase instance, embedded with Cursor’s own model. Copying an existing namespace for a new checkout cut the median time to first query from 7.87 seconds to 525 milliseconds, and the p99 from 4.03 hours to 21 seconds (",{"tag":159,"href":55,"children":240},[241],"turbopuffer","). That is the onboarding problem every code index has: the first index of a large repository is the slow part.",{"type":134,"content":244},[245,246,248,249,252,253,256,257,209],"Kilo Code ships the same pattern for a local setup: LanceDB by default or Qdrant as an external store, embedding providers including Ollama for a fully local stack, file watching plus hash-based caching for incremental updates, and handling for branch switches. Its defaults include a minimum search score of 0.4 and up to 50 results (",{"tag":159,"href":58,"children":247},[196],"). Cline itself, by contrast, currently documents built-in tools such as ",{"tag":173,"children":250},[251],"read_files"," and ",{"tag":173,"children":254},[255],"search_codebase"," and no embedding index (",{"tag":159,"href":258,"children":259},"https:\u002F\u002Fdocs.cline.bot\u002Ftools-reference\u002Fall-cline-tools.md",[260],"Cline docs",{"type":164,"src":262,"alt":263,"wide":167,"caption":264},"\u002Fimages\u002Fblog\u002Fcode-rag-repository-indexing\u002Fdiagram-19.webp","Indexed retrieval path for code Five steps from left to right: the repository tracked by Merkle hashes; tree-sitter turns changed files into AST chunks; chunks are embedded, with embeddings cached by content hash; vectors go into a store with one namespace per codebase; search returns chunks, and the agent then reads the files.","The indexed path pays once for structure and embeddings; the agentic path (grep, glob, read, repeat) skips every box here and pays in tokens per search instead.",{"type":266,"head":267,"rows":278},"table",[268,270,272,274,276],[269],"System",[271],"Chunking",[273],"Retrieval primitive",[275],"Storage",[277],"Freshness",[279,290,300,310,320],[280,282,284,286,288],[281],"Aider repo map",[283],"tree-sitter tags, signatures only",[285],"graph ranking over file dependencies",[287],"none; map goes into the prompt",[289],"rebuilt per request, 1k-token default",[291,292,294,296,298],[235],[293],"syntactic chunks",[295],"semantic search over embeddings",[297],"turbopuffer, namespace per codebase",[299],"Merkle tree, only changed branches synced",[301,302,304,306,308],[196],[303],"tree-sitter blocks, 100–1,000 chars",[305],"vector search, min score 0.4",[307],"LanceDB or Qdrant",[309],"file watcher plus hash cache",[311,312,314,316,318],[222],[313],"none; symbols from language servers",[315],"LSP definition and reference lookup",[317],"language server state",[319],"live",[321,323,325,327,328],[322],"Claude Code",[324],"none",[326],"grep (ripgrep) and glob, just in time",[324],[329],"never stale; tokens per search",{"type":150,"level":151,"id":121,"text":122},{"type":134,"content":332},[333,334,337,338,341,342,345,346,209],"As of October 2026, there are three practical routes. Hosted and code-tuned: Voyage lists ",{"tag":173,"children":335},[336],"voyage-code-4"," as its current code model at $0.12 per million tokens with 200 million free tokens, while ",{"tag":173,"children":339},[340],"voyage-code-3"," has moved to the older models at $0.18 (",{"tag":159,"href":64,"children":343},[344],"Voyage","). Self-hosted: Qwen3-Embedding comes in 0.6B, 4B and 8B sizes with a 32k context, Matryoshka dimensions and support for more than 100 languages including programming languages (",{"tag":159,"href":67,"children":347},[348],"Qwen",{"type":134,"content":350},[351,352,356,357,361,362,366],"The third route is a general multimodal or long-context model. jina-embeddings-v4 can emit either a single 2048-dimensional vector or token-level vectors for late interaction, reports 71.59 on the CoIR code benchmark, and carries the Qwen Research License, which matters for commercial use (",{"tag":159,"href":353,"children":354},"https:\u002F\u002Fjina.ai\u002Fmodels\u002Fjina-embeddings-v4\u002F",[355],"Jina","). Cohere Embed v4 offers a 128k context and Matryoshka output from 256 to 1536 dimensions at $0.12 per million tokens (",{"tag":159,"href":358,"children":359},"https:\u002F\u002Fdocs.cohere.com\u002Fchangelog\u002Fembed-multimodal-v4",[360],"Cohere","). Which model to pick is a general question, covered in ",{"tag":144,"to":363,"children":364},"\u002Fblog\u002Fembedding-model-selection-2026",[365],"embedding model selection","; for code, the long context matters because whole files and long functions fit.",{"type":134,"content":368},[369],"For a repository of a few hundred thousand lines, the embedding bill is small at these prices; the real costs are re-embedding after churn and the engineering to keep the index correct. Code-tuning pays most when queries are in natural language and the code uses domain names the general model has never seen.",{"type":150,"level":151,"id":124,"text":125},{"type":134,"content":372},[373,374,377],"Measure on your own repository, with public benchmarks as a sanity check. CoIR, published at ACL 2025, combines ten datasets across eight retrieval tasks and seven domains, and its authors found that even state-of-the-art models had significant difficulties with code retrieval (",{"tag":159,"href":70,"children":375},[376],"CoIR","). Your queries, identifiers and conventions are not in it, so build a small set yourself.",{"type":379,"ordered":167,"items":380},"list",[381,383,385,387],[382],"Collect 30 to 50 real questions from tickets and code review, each with the file and symbol that answers it.",[384],"Mix natural-language questions with exact-name lookups, since they stress different retrievers.",[386],"Score recall at 5 and 10 for each chunking and model variant, and keep the set in the repository.",[388],"Re-run it after model changes and large refactors, not only once.",{"type":134,"content":390},[391,392,395,396,228],"Freshness needs as much design as quality. The working patterns are file watchers with content-hash caches (Kilo Code), Merkle trees that sync only changed branches (Cursor), and memoized pipelines: CocoIndex keys its cache per file and per processing step, so a commit that touches a few files re-embeds only those, deleted files remove their rows, and the index runs either as a catch-up pass in CI or in live mode watching the working tree with about one second of latency (",{"tag":159,"href":73,"children":393},[394],"CocoIndex","). Deletions are the case most setups get wrong, which is how an agent ends up citing removed code. Late chunking and contextual retrieval add context to chunks on top of this, see ",{"tag":144,"to":397,"children":398},"\u002Fblog\u002Fcontextual-retrieval-late-chunking",[399],"contextual retrieval and late chunking",{"type":401,"variant":402,"title":403,"body":404},"callout","warn","Leaderboards are not your repository",[405],[406],"Benchmark scores for code embeddings change from month to month, and none of them contain your naming conventions. Treat a CoIR number as a filter for the shortlist and let your own question set decide.",{"type":150,"level":151,"id":127,"text":128},{"type":134,"content":409},[410,411,414,415,209],"Grep beats an index when the repository is small enough to search quickly, changes constantly, or cannot leave the machine. Anthropic describes Claude Code as a hybrid: CLAUDE.md is loaded up front, and “primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing and complex syntax trees.” The same post notes that runtime exploration is slower than pre-computed retrieval (",{"tag":159,"href":49,"children":412},[413],"Anthropic","). Claude Code’s Grep tool is built on ripgrep, alongside Glob (",{"tag":159,"href":416,"children":417},"https:\u002F\u002Fcode.claude.com\u002Fdocs\u002Fen\u002Ftools-reference",[418],"Claude Code docs",{"type":134,"content":420},[421],"I run Claude Code, opencode and Codex side by side, and the trade-off is visible in daily use: grep-first agents spend tokens on several search-and-read rounds in an unfamiliar area, and never return code that no longer exists. The cost of an index is the opposite: storage, embedding spend and the freshness machinery above, in exchange for fewer rounds on questions phrased by meaning.",{"type":379,"ordered":423,"items":424},false,[425,430,435,440,445],[426,429],{"tag":137,"children":427},[428],"Repository size",": below a size where ripgrep answers instantly, start with grep and a repo map.",[431,434],{"tag":137,"children":432},[433],"Churn",": high churn favors grep or LSP; an index needs incremental updates that handle deletions.",[436,439],{"tag":137,"children":437},[438],"Monorepo",": many teams and unfamiliar areas favor an index, ideally shared and reused across checkouts.",[441,444],{"tag":137,"children":442},[443],"Compliance",": if code cannot leave the machine, use local embedders and a local store, or skip the index.",[446,449],{"tag":137,"children":447},[448],"Hybrid by default",": grep for exact names, LSP for definitions and references, an index for meaning.",{"type":134,"content":451},[452,453,457],"If you are building retrieval for a coding agent or an internal code assistant, my ",{"tag":144,"to":454,"children":455},"\u002Fexpertise\u002Fai-engineer",[456],"AI engineering work"," covers this kind of setup, from chunking to evaluation.",{"type":150,"level":151,"id":130,"text":131},{"type":379,"ordered":167,"items":460},[461,464,467,470,473,476,479,482,485,488,491,494],[462],{"tag":159,"href":40,"children":463},[39],[465],{"tag":159,"href":43,"children":466},[42],[468],{"tag":159,"href":46,"children":469},[45],[471],{"tag":159,"href":49,"children":472},[48],[474],{"tag":159,"href":52,"children":475},[51],[477],{"tag":159,"href":55,"children":478},[54],[480],{"tag":159,"href":58,"children":481},[57],[483],{"tag":159,"href":61,"children":484},[60],[486],{"tag":159,"href":64,"children":487},[63],[489],{"tag":159,"href":67,"children":490},[66],[492],{"tag":159,"href":70,"children":493},[69],[495],{"tag":159,"href":73,"children":496},[72],[498,571,628,686],{"slug":499,"published":500,"minutes":501,"category":7,"tags":502,"keywords":506,"about":519,"sources":529,"cover":565,"og":566,"expertise":76,"locales":567,"lang":78,"title":568,"description":569,"coverAlt":570},"embedding-model-selection-2026","2026-10-17",9,[12,9,503,504,505],"MTEB","Vector search","Model selection",[507,508,509,510,511,512,513,514,515,516,517,518],"best embedding model 2026","how to choose an embedding model","embedding model comparison","MTEB leaderboard explained","RTEB benchmark","open source embedding model","multilingual embedding model","voyage-4 vs openai embeddings","Qwen3-Embedding","EmbeddingGemma","embedding model for RAG","switching embedding models re-embed",[520,523,526],{"name":521,"url":522},"Word embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FWord_embedding",{"name":524,"url":525},"Information retrieval","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FInformation_retrieval",{"name":527,"url":528},"Discounted cumulative gain","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDiscounted_cumulative_gain",[530,533,536,539,542,545,548,550,553,556,559,562],{"title":531,"url":532},"Enevoldsen et al.: MMTEB, Massive Multilingual Text Embedding Benchmark (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2502.13595",{"title":534,"url":535},"Hugging Face: Introducing RTEB, a new standard for retrieval evaluation (Oct 2025)","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Frteb",{"title":537,"url":538},"OpenAI: Vector embeddings guide","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fembeddings",{"title":540,"url":541},"OpenAI: API pricing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fpricing",{"title":543,"url":544},"Voyage AI: The Voyage 4 model family, shared embedding space with MoE architecture (Jan 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F01\u002F15\u002Fvoyage-4\u002F",{"title":546,"url":547},"Voyage AI: Text embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fembeddings",{"title":549,"url":64},"Voyage AI: Pricing",{"title":551,"url":552},"Google: Gemini API embeddings documentation","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fembeddings",{"title":554,"url":555},"Google: Gemini API pricing","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fpricing",{"title":557,"url":558},"Cohere: Embed models documentation","https:\u002F\u002Fdocs.cohere.com\u002Fdocs\u002Fcohere-embed",{"title":560,"url":561},"Qwen: Qwen3-Embedding-8B model card","https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-Embedding-8B",{"title":563,"url":564},"Google Developers Blog: Introducing EmbeddingGemma (Sep 2025)","https:\u002F\u002Fdevelopers.googleblog.com\u002Fen\u002Fintroducing-embeddinggemma\u002F","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fog.jpg",[78,79,80],"Choosing an embedding model in 2026: leaderboards, costs and a bake-off on your own data","How to pick an embedding model for RAG in 2026: what MTEB and RTEB tell you, hosted vs open weights, prices, dimensions, and a one-week bake-off.","Three columns of stacked blocks, the middle tallest, each measured by a short bar and a wide bracket.",{"slug":572,"published":500,"minutes":573,"category":7,"tags":574,"keywords":577,"about":589,"sources":597,"cover":622,"og":623,"expertise":76,"locales":624,"lang":78,"title":625,"description":626,"coverAlt":627},"contextual-retrieval-late-chunking",10,[9,271,12,575,576],"Contextual retrieval","Late chunking",[578,579,580,581,582,583,584,585,586,587,588],"contextual retrieval","anthropic contextual retrieval","late chunking","contextual chunk embeddings","contextual embeddings rag","contextual bm25","chunk context rag","voyage-context-3","voyage-context-4","late chunking vs contextual retrieval","rag chunking context loss",[590,591,594],{"name":27,"url":28},{"name":592,"url":593},"Okapi BM25","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOkapi_BM25",{"name":595,"url":596},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",[598,601,604,607,610,613,616,619],{"title":599,"url":600},"Anthropic: Introducing Contextual Retrieval (Sep 2024)","https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fcontextual-retrieval",{"title":602,"url":603},"Günther et al.: Late Chunking, Contextual Chunk Embeddings Using Long-Context Embedding Models (2024, rev. 2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.04701",{"title":605,"url":606},"Jina AI: Late Chunking in Long-Context Embedding Models (Aug 2024)","https:\u002F\u002Fjina.ai\u002Fnews\u002Flate-chunking-in-long-context-embedding-models\u002F",{"title":608,"url":609},"Voyage AI: Introducing voyage-context-3 (Jul 2025)","https:\u002F\u002Fblog.voyageai.com\u002F2025\u002F07\u002F23\u002Fvoyage-context-3\u002F",{"title":611,"url":612},"Voyage AI: voyage-context-4 (Jun 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F06\u002F29\u002Fvoyage-context-4\u002F",{"title":614,"url":615},"Voyage AI: Contextualized chunk embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fcontextualized-chunk-embeddings",{"title":617,"url":618},"Anthropic: Claude API pricing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fpricing",{"title":620,"url":621},"Qu, Tu and Bao: Is Semantic Chunking Worth the Computational Cost? (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13070","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fog.jpg",[78,79,80],"Contextual retrieval and late chunking: giving RAG chunks back their context","Chunks lose their document. How contextual retrieval, late chunking and contextualized embedding models fix that, what each costs, and when to skip it.","Four loose square pieces gathered by a wide bracket below, each joined upward to a large sheet.",{"slug":629,"published":500,"minutes":501,"category":7,"tags":630,"keywords":634,"about":647,"sources":655,"cover":680,"og":681,"expertise":76,"locales":682,"lang":78,"title":683,"description":684,"coverAlt":685},"rag-access-control-permissions",[9,631,632,633,504],"Access control","Security","Multi-tenancy",[635,636,637,638,639,640,641,642,643,644,645,646],"rag access control","permission aware rag","document level security rag","rag multi tenancy","vector database access control","pgvector row level security","security trimming azure ai search","openfga rag authorization","qdrant multitenancy","pinecone namespaces multitenancy","owasp llm08 vector and embedding weaknesses","rag data leakage",[648,651,654],{"name":649,"url":650},"Access-control list","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FAccess-control_list",{"name":652,"url":653},"Multitenancy","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMultitenancy",{"name":27,"url":28},[656,659,662,665,668,671,674,677],{"title":657,"url":658},"OWASP GenAI Security Project: LLM08:2025 Vector and Embedding Weaknesses","https:\u002F\u002Fgenai.owasp.org\u002Fllmrisk\u002Fllm082025-vector-and-embedding-weaknesses\u002F",{"title":660,"url":661},"PostgreSQL documentation: Row Security Policies","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fddl-rowsecurity.html",{"title":663,"url":664},"pgvector README: filtering and iterative index scans","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector",{"title":666,"url":667},"Qdrant: Multitenancy and partitions guide","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fguides\u002Fmultiple-partitions\u002F",{"title":669,"url":670},"Pinecone: Implement multitenancy using namespaces","https:\u002F\u002Fdocs.pinecone.io\u002Fguides\u002Findex-data\u002Fimplement-multitenancy",{"title":672,"url":673},"Microsoft Learn: Document-level access control in Azure AI Search","https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fsearch-document-level-access-overview",{"title":675,"url":676},"OpenFGA: RAG authorization use case","https:\u002F\u002Fopenfga.dev\u002Fdocs\u002Fuse-cases\u002Frag-authorization",{"title":678,"url":679},"Morris et al.: Text Embeddings Reveal (Almost) As Much As Text (EMNLP 2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.06816","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fog.jpg",[78,79,80],"Access control for RAG: permission-aware retrieval without leaking documents","How RAG assistants leak documents and how to stop it: ACLs on every chunk, pre-filtering, row-level security in pgvector, permission sync and leak tests.","Seven cards in a row, six with a lock badge, one bare card sending an arrow outward.",{"slug":687,"published":500,"minutes":573,"category":7,"tags":688,"keywords":693,"about":705,"sources":713,"cover":740,"og":741,"expertise":742,"locales":743,"lang":78,"title":744,"description":745,"coverAlt":746},"rag-product-catalog-pim-data",[9,689,690,691,692],"PIM","Product data","B2B e-commerce","Metadata filtering",[694,695,696,697,698,699,700,701,702,703,704],"rag product catalog","rag pim data","llm product data","product catalog chatbot","akeneo rag","pimcore ai search","rag structured data","metadata filtering rag","product attributes embeddings","b2b product assistant","etim classification ai",[706,709,710],{"name":707,"url":708},"Product information management","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FProduct_information_management",{"name":27,"url":28},{"name":711,"url":712},"ETIM (standard)","https:\u002F\u002Fwww.etim-international.com\u002Fclassification\u002F",[714,717,720,723,725,728,731,734,737],{"title":715,"url":716},"Akeneo API: Products, product values, metric values and product models","https:\u002F\u002Fapi.akeneo.com\u002Fconcepts\u002Fproducts.html",{"title":718,"url":719},"Akeneo Help Center: What about products with variants?","https:\u002F\u002Fhelp.akeneo.com\u002Fserenity-take-the-power-over-your-products\u002Fserenity-what-about-products-with-variants",{"title":721,"url":722},"Pimcore documentation: Classification Store","https:\u002F\u002Fdocs.pimcore.com\u002Fplatform\u002FPimcore\u002FObjects\u002FObject_Classes\u002FData_Types\u002FClassification_Store\u002F",{"title":724,"url":712},"ETIM International: Classification",{"title":726,"url":727},"ETIM International: ETIM releases and the IXF format","https:\u002F\u002Fetim-international.com\u002F?p=754",{"title":729,"url":730},"Reddy et al.: Shopping Queries Dataset, a large-scale ESCI benchmark for improving product search (2022)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2206.06588",{"title":732,"url":733},"Amazon Science: esci-data repository","https:\u002F\u002Fgithub.com\u002Famazon-science\u002Fesci-data",{"title":735,"url":736},"Qdrant documentation: Filtering","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fconcepts\u002Ffiltering\u002F",{"title":738,"url":739},"Sui et al.: Table Meets LLM, can large language models understand structured table data? (WSDM 2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.13062","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fog.jpg","b2b-ecommerce-developer",[78,79,80],"RAG over product catalogues: turning PIM data into answers that hold","Product data is structured, so chunking it like PDFs fails. How to serialize PIM attributes, route spec questions to filters and keep answers exact.","A frame of nine squares with tiny symbols, funnelling down to one card.",1792337027762]