[{"data":1,"prerenderedAt":736},["ShallowReactive",2],{"blog-incremental-indexing-cdc-en":3},{"slug":4,"published":5,"updated":6,"minutes":7,"category":8,"tags":9,"keywords":15,"about":26,"sources":35,"cover":72,"og":73,"expertise":74,"locales":75,"lang":76,"title":79,"description":80,"coverAlt":81,"metaTitle":82,"takeaways":83,"faq":89,"toc":105,"blocks":133,"others":486},"incremental-indexing-cdc","2026-09-25","2026-10-17",8,"rag",[10,11,12,13,14],"RAG","Change data capture","PostgreSQL","Vector databases","Data freshness",[16,17,18,19,20,21,22,23,24,25],"incremental indexing for RAG","change data capture RAG","Debezium Postgres logical decoding","re-embedding cost","blue\u002Fgreen index swap vector database","index staleness monitoring","delete propagation vector index","permission changes RAG index","Qdrant aliases","high water mark change detection",[27,29,32],{"name":11,"url":28},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FChange_data_capture",{"name":30,"url":31},"Retrieval-augmented generation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRetrieval-augmented_generation",{"name":33,"url":34},"Blue–green deployment","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBlue%E2%80%93green_deployment",[36,39,42,45,48,51,54,57,60,63,66,69],{"title":37,"url":38},"PostgreSQL docs: logical decoding concepts","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Flogicaldecoding-explanation.html",{"title":40,"url":41},"Debezium docs: PostgreSQL connector","https:\u002F\u002Fdebezium.io\u002Fdocumentation\u002Freference\u002Fstable\u002Fconnectors\u002Fpostgresql.html",{"title":43,"url":44},"PostgreSQL docs: CREATE PUBLICATION","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fsql-createpublication.html",{"title":46,"url":47},"PostgreSQL docs: pg_replication_slots","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fview-pg-replication-slots.html",{"title":49,"url":50},"Azure AI Search: changed and deleted blobs","https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fsearch-howto-index-changed-deleted-blobs",{"title":52,"url":53},"Azure AI Search: scheduling indexers","https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fsearch-howto-schedule-indexers",{"title":55,"url":56},"Qdrant docs: collections and aliases","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fconcepts\u002Fcollections\u002F",{"title":58,"url":59},"Elastic docs: aliases","https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fmanage-data\u002Fdata-store\u002Faliases",{"title":61,"url":62},"pgvector on GitHub","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector",{"title":64,"url":65},"GDPR Article 17","https:\u002F\u002Fgdpr-info.eu\u002Fart-17-gdpr\u002F",{"title":67,"url":68},"OpenAI API pricing","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fpricing",{"title":70,"url":71},"Voyage AI docs: pricing","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fpricing","\u002Fimages\u002Fblog\u002Fincremental-indexing-cdc\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fincremental-indexing-cdc\u002Fog.jpg","ai-engineer",[76,77,78],"en","de","hu","Keeping a RAG index fresh: incremental indexing, CDC and blue\u002Fgreen re-embedding","Keeping a RAG index current: staleness budgets, dirty tracking, CDC from Postgres with Debezium, deletes and permissions, blue\u002Fgreen swaps and costs.","A database streams changes into a cabinet, with a second cabinet and a switch beside it.","Keeping a RAG index fresh · Balázs Csorba",[84,85,86,87,88],"Give every source a staleness budget in minutes and measure lag against it; “eventually consistent” is not a requirement anyone can test.","Watermarks and content hashes are enough for many sources; CDC from Postgres via logical decoding and Debezium is for sources where minutes matter, and its replication slots must be monitored.","Deletes and permission changes are the deltas indexers miss: Azure AI Search documents that indexers do not track deletions, and GDPR erasure must happen without undue delay.","Re-embedding for a model upgrade belongs in a new collection behind an alias, switched with one API call and kept for rollback; in pgvector, build indexes concurrently.","The tokens are the cheap part: re-embedding 100 million tokens costs about $2 at $0.02 per million; engineering, backfill time and correctness are the real cost.",[90,93,96,99,102],{"q":91,"a":92},"How do I keep a RAG index up to date?","Track which source records changed, by watermark, content hash or change data capture, re-chunk and re-embed only those, propagate deletes and permission changes explicitly, and run a periodic sweep that catches anything the event path missed. Measure the lag against a staleness budget per source.",{"q":94,"a":95},"What is change data capture for RAG?","Reading changes from the database’s log instead of polling tables. In Postgres, logical decoding exposes committed inserts, updates and deletes, and Debezium turns them into events that an indexer can consume within seconds. It needs replication slots, which retain WAL until consumed and must be monitored.",{"q":97,"a":98},"How do deletes reach the vector index?","Only if you send them. Many indexers do not track deletion in the source, so use delete events from CDC, a soft-delete flag the indexer reads, or a periodic reconciliation that removes chunks whose source id no longer exists. Personal data under GDPR must be erased without undue delay.",{"q":100,"a":101},"How do I re-embed with a new model without downtime?","Build a new collection with the new model while the old one serves, backfill it and dual-write new changes to both, then switch an alias to the new collection in one call and keep the old one read-only for rollback. In pgvector, create the new index concurrently and switch in configuration.",{"q":103,"a":104},"How much does re-embedding a corpus cost?","At embedding API prices of $0.02 to $0.13 per million tokens as of October 2026, re-embedding 100 million tokens costs roughly $2 to $13. The real costs are the backfill time, rate limits, the engineering of the swap, and re-checking retrieval quality on the new model.",[106,109,112,115,118,121,124,127,130],{"id":107,"title":108},"staleness-budget","What fresh means: a staleness budget",{"id":110,"title":111},"dirty-tracking","Dirty tracking with watermarks and hashes",{"id":113,"title":114},"cdc-from-postgres","Change data capture from Postgres",{"id":116,"title":117},"deletes-and-permissions","Deletes and permission changes",{"id":119,"title":120},"reindex-and-blue-green-swaps","Full reindex and blue\u002Fgreen swaps",{"id":122,"title":123},"cost-of-re-embedding","The cost of re-embedding",{"id":125,"title":126},"when-not-to-use-cdc","When CDC is overkill",{"id":128,"title":129},"checklist","A checklist for an index that stays current",{"id":131,"title":132},"sources","Sources",[134,142,151,154,157,160,161,174,182,185,186,201,207,222,224,243,244,254,260,267,268,278,282,292,311,312,338,341,344,345,352,417,418,435,447,448],{"type":135,"content":136},"paragraph",[137,141],{"tag":138,"children":139},"strong",[140],"Incremental indexing for RAG"," is the layer that keeps the index true after the first crawl: new documents get chunked and embedded, changed ones get replaced, deleted ones disappear, and permission changes reach the index before someone sees a document they should not. Most RAG prototypes skip it because the first demo uses a static folder. Production corpora change all day, and an assistant that cites last week’s price list or a revoked policy is worse than one that says it does not know.",{"type":135,"content":143},[144,145,150],"This article covers a staleness budget, dirty tracking with watermarks and hashes, change data capture from Postgres, deletes and permission changes, full reindexing with blue\u002Fgreen swaps, the cost of re-embedding and when CDC is overkill. Chunking, retrieval and reranking are covered in ",{"tag":146,"to":147,"children":148},"link","\u002Fblog\u002Frag-pipeline-chunking-hybrid-search-reranking",[149],"RAG pipelines: chunking, hybrid search and reranking","; versions and prices are as of October 2026.",{"type":152,"level":153,"id":107,"text":108},"heading",2,{"type":135,"content":155},[156],"Fresh means a measured number per source, not a feeling. Write down, for each source, how many minutes a change may take to reach the index: perhaps five for stock levels, an hour for product descriptions, a day for archived manuals, and immediately for revoked access. No standard prescribes these numbers; they come from what a wrong answer costs. Then measure the actual lag from source commit to index write, and alert when it exceeds the budget.",{"type":135,"content":158},[159],"The budget also tells you which mechanism you need. A daily budget is met by a nightly sweep; a five-minute budget needs either a frequent scan or events; a budget of seconds for permissions usually means checking access at query time as well as in the index. Splitting the index helps here. In Spryker and Pimcore catalogs, price and stock change all day while long descriptions change rarely, so volatile scalar fields can be updated as metadata without re-embedding the text they sit next to.",{"type":152,"level":153,"id":110,"text":111},{"type":135,"content":162},[163,164,168,169,173],"Most sources do not need a database log, only a reliable way to find what changed since the last run. A high-water-mark policy is the classic pattern: remember the largest ",{"tag":165,"children":166},"code",[167],"updated_at"," value processed and fetch rows above it next time. Azure AI Search names it as a reusable change-detection policy, and its scheduled indexers run between every 5 minutes and every 24 hours (",{"tag":170,"href":53,"children":171},"a",[172],"Azure AI Search","). Use a monotonic column, keep a small overlap window and make writes idempotent, because clocks and long transactions produce rows that commit with an older timestamp.",{"type":135,"content":175},[176,177,181],"Content hashes complement watermarks. Hash the text that will be embedded, not the whole row, and skip re-embedding when the hash is unchanged; a price update then touches metadata only. The same idea scales down to files and repositories, where a hash per file decides what to re-chunk, as described in ",{"tag":146,"to":178,"children":179},"\u002Fblog\u002Fcode-rag-repository-indexing",[180],"RAG over code",". The weak spot of both is deletion: a row that no longer exists never shows up in a query for changed rows.",{"type":135,"content":183},[184],"A sweep also needs a backstop for drift. Once a week, compare the full list of source ids with the ids present in the index: anything in the index but not in the source is a missed delete, and anything in the source but not in the index is a missed insert. That comparison is cheap, it catches bugs in every other mechanism, and its result is a number you can put on the same dashboard as the lag.",{"type":152,"level":153,"id":113,"text":114},{"type":135,"content":187},[188,189,191,192,195,196,200],"Change data capture reads committed changes from the database’s write-ahead log instead of polling tables, so inserts, updates and deletes all arrive as events within seconds. In Postgres the mechanism is logical decoding, “the process of extracting all persistent changes to a database’s tables into a coherent, easy to understand format”, which relies on replication slots and an output plugin (",{"tag":170,"href":38,"children":190},[12],"). A publication defines which tables are in scope (",{"tag":170,"href":44,"children":193},[194],"CREATE PUBLICATION","), but schema changes are not replicated, so DDL is a separate job (",{"tag":170,"href":197,"children":198},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Flogical-replication-restrictions.html",[199],"restrictions",").",{"type":202,"src":203,"alt":204,"wide":205,"caption":206},"image","\u002Fimages\u002Fblog\u002Fincremental-indexing-cdc\u002Fdiagram-11.webp","Event lane for a fresh index Five steps from left to right: a change is committed in Postgres; logical decoding exposes it through a replication slot; Debezium turns it into an insert, update or delete event; a transform filters relevant fields, re-chunks and embeds changed text; and the index upserts or deletes the affected chunks. A scheduled sweep by updated_at catches anything the event lane missed, and a lag alarm watches the replication slot.",true,"The event lane delivers changes in seconds; a slower sweep and a lag alarm make sure nothing stays wrong for long.",{"type":135,"content":208},[209,210,213,214,217,218,221],"Debezium is the common way to consume it. Its PostgreSQL connector needs ",{"tag":165,"children":211},[212],"wal_level=logical"," and enough WAL senders and replication slots, and its documentation is blunt about the risk: replication slots retain all WAL that Debezium needs even during outages, so they must be monitored to avoid disk exhaustion and catalog bloat (",{"tag":170,"href":41,"children":215},[216],"Debezium","). Update events include the previous column values only with ",{"tag":165,"children":219},[220],"REPLICA IDENTITY FULL","; a heartbeat interval keeps the slot advancing when the captured tables are quiet, and a signaling table triggers targeted backfills instead of a full reindex.",{"type":165,"code":223},"-- Postgres side of CDC: scope, slot health and lag\nALTER TABLE products REPLICA IDENTITY FULL;            -- update events carry old values too\nCREATE PUBLICATION rag_sources FOR TABLE products, product_texts, document_acl;\n\n-- WAL retained by each slot: alert long before the disk fills\nSELECT slot_name, active, wal_status,\n       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS behind\nFROM pg_replication_slots;",{"type":135,"content":225},[226,227,230,231,234,235,237,238,242],"Postgres exposes the numbers you need for the lag alarm: ",{"tag":165,"children":228},[229],"pg_replication_slots"," shows each slot’s confirmed position and WAL status, and ",{"tag":165,"children":232},[233],"max_slot_wal_keep_size"," caps how much WAL a slot may hold before it is invalidated (",{"tag":170,"href":47,"children":236},[12],"). Alert on the gap between the current WAL position and the slot’s confirmed position long before it reaches that cap. For order and catalog data that already flows between systems, the same outbox and idempotency patterns apply, see ",{"tag":146,"to":239,"children":240},"\u002Fblog\u002Fb2b-order-sync-erp",[241],"order sync between shop and ERP",".",{"type":152,"level":153,"id":116,"text":117},{"type":135,"content":245},[246,247,249,250,253],"Deletes and permission changes are the two deltas that indexers most often miss, and they are the two with legal consequences. Azure AI Search states it plainly: an indexer does not track object deletion in data sources, so a soft-delete strategy is needed, and the policy must be applied from the very first indexer run or deleted documents stay in the index (",{"tag":170,"href":50,"children":248},[172],"). Under GDPR Article 17 personal data must be erased without undue delay once erasure is due (",{"tag":170,"href":65,"children":251},[252],"GDPR","), and a vector index holding chunks of that data is part of the scope.",{"type":135,"content":255},[256,257,200],"Handle both explicitly. Store the source id and document id on every chunk so a delete can remove all chunks of a document in one call; consume delete events from CDC where you have them; and run a periodic reconciliation that removes chunks whose source no longer exists. For access control, keep the permission state in the index next to the text and vectors, and propagate permission changes as their own events. Azure AI Search now stores ACL metadata inside the index and added per-run incremental ACL sync for its SharePoint indexer in 2026 (",{"tag":170,"href":258,"children":259},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fwhats-new",[172],{"type":261,"variant":262,"title":263,"body":264},"callout","warn","A deleted document is still answerable until you delete its chunks",[265],[266],"Removing a document from the source system does not remove it from the vector index, the cache in front of it or any answer log. Test erasure end to end: delete a known document, then ask the assistant a question only that document could answer, and check that the citation is gone.",{"type":152,"level":153,"id":119,"text":120},{"type":135,"content":269},[270,271,274,275,200],"Some changes cannot be applied incrementally: a new embedding model, a new chunking strategy or a new index configuration means every vector changes. Treat that as a migration with a blue\u002Fgreen swap. Qdrant documents aliases for exactly this case, switching between versions of vectors “when upgrading to a new version of the neural network”, with alias changes sent as one actions request (",{"tag":170,"href":56,"children":272},[273],"Qdrant","); Elasticsearch aliases serve the same purpose and are documented as a way to reindex without downtime (",{"tag":170,"href":59,"children":276},[277],"Elastic",{"type":202,"src":279,"alt":280,"wide":205,"caption":281},"\u002Fimages\u002Fblog\u002Fincremental-indexing-cdc\u002Fdiagram-21.webp","Blue\u002Fgreen re-embedding timeline Four phases from left to right on one timeline. Build: a new collection v2 is created with the new model while v1 serves all traffic. Backfill: v2 is filled from the source and new changes are written to both. Switch: one alias call points the application from v1 to v2. Keep: v1 stays read-only for rollback, then it is dropped. Below, the pgvector variant: create the new index concurrently, then switch in application configuration.","A model upgrade is a migration: build beside, backfill, switch with one call, and keep the old index until the new one has proven itself.",{"type":135,"content":283},[284,285,288,289,200],"The sequence is the same everywhere: build the new collection with the new model while the old one serves, backfill it from the source, write new changes to both during the backfill, switch the alias, and keep the old collection read-only until the new one has passed your retrieval evaluation in production. The documentation describes the alias call, not its transactional guarantees under load, so test the switch on a staging copy first. In pgvector, the equivalent is a new column or table, an index built with ",{"tag":165,"children":286},[287],"CREATE INDEX CONCURRENTLY"," so writes are not blocked, and a switch in application configuration (",{"tag":170,"href":62,"children":290},[291],"pgvector",{"type":135,"content":293},[294,295,298,299,302,303,306,307,242],"pgvector adds two practical notes: index builds are much faster when the HNSW graph fits into ",{"tag":165,"children":296},[297],"maintenance_work_mem",", and the defaults of ",{"tag":165,"children":300},[301],"m = 16"," and ",{"tag":165,"children":304},[305],"ef_construction = 64"," are a reasonable start for the new index. Choosing the model you migrate to is a separate decision, covered in ",{"tag":146,"to":308,"children":309},"\u002Fblog\u002Fembedding-model-selection-2026",[310],"embedding model selection",{"type":152,"level":153,"id":122,"text":123},{"type":135,"content":313},[314,315,318,319,322,323,326,327,318,330,333,334,337],"The tokens are the cheap part. As of October 2026, OpenAI lists ",{"tag":165,"children":316},[317],"text-embedding-3-small"," at $0.02 and ",{"tag":165,"children":320},[321],"text-embedding-3-large"," at $0.13 per million tokens (",{"tag":170,"href":68,"children":324},[325],"OpenAI","), and Voyage lists ",{"tag":165,"children":328},[329],"voyage-4-lite",{"tag":165,"children":331},[332],"voyage-4"," at $0.06 per million, with a batch discount (",{"tag":170,"href":71,"children":335},[336],"Voyage","). Re-embedding a corpus of 100 million tokens therefore costs roughly $2 to $13 in API fees. That is arithmetic on list prices, not a quote, and it is small next to everything else.",{"type":135,"content":339},[340],"The real costs are elsewhere: hours of backfill under rate limits, a dual-write window that must be correct, extra storage while two indexes exist, and a retrieval evaluation on the new model before the switch. On my own Nuxt site, where I run LLM features over a small corpus, re-embedding everything is trivial, and that is exactly why the token bill is not a reason to build CDC; freshness, deletes and permissions are.",{"type":135,"content":342},[343],"One shortcut is worth checking before a full re-embed. Some vendors describe model families whose members share an embedding space, which would allow upgrading the query side first; verify such a claim in the vendor’s own documentation and with your evaluation set before relying on it, because mixing vectors from incompatible models silently degrades retrieval.",{"type":152,"level":153,"id":125,"text":126},{"type":135,"content":346},[347,348,200],"CDC is overkill when the staleness budget is hours or a day, when the source is a SaaS system without database access, or when nobody on the team will watch replication slots. A nightly or hourly sweep with watermarks, content hashes and a deletion reconciliation is simpler, easier to reason about and enough for most document collections. For SaaS sources, webhooks are the event path: verify a secret, respond within 10 seconds, check the event type, redeliver missed deliveries and deduplicate on the delivery id, as GitHub’s guidance describes (",{"tag":170,"href":349,"children":350},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Fwebhooks\u002Fusing-webhooks\u002Fbest-practices-for-using-webhooks",[351],"GitHub",{"type":353,"head":354,"rows":365},"table",[355,357,359,361,363],[356],"Mechanism",[358],"Typical staleness",[360],"Catches deletes",[362],"Catches permission changes",[364],"Operational load",[366,376,387,397,408],[367,369,371,373,374],[368],"Nightly full reindex",[370],"up to a day",[372],"yes, by rebuild",[372],[375],"low, but costly at scale",[377,379,381,383,385],[378],"updated_at sweep",[380],"minutes to hours",[382],"no, needs reconciliation",[384],"only if ACL rows have timestamps",[386],"low",[388,390,392,394,396],[389],"Content-hash diff",[391],"per run",[393],"yes, if the full list is compared",[395],"no",[386],[398,400,402,404,406],[399],"CDC via logical decoding",[401],"seconds",[403],"yes, delete events",[405],"yes, if ACL tables are published",[407],"high: slots, lag, schema changes",[409,411,412,414,415],[410],"Inbound webhooks",[401],[413],"if the source sends them",[413],[416],"medium: retries, dedupe",{"type":152,"level":153,"id":128,"text":129},{"type":419,"ordered":205,"items":420},"list",[421,423,425,427,429,431,433],[422],"Set a staleness budget per source and measure lag against it.",[424],"Store source id, document id, content hash and permissions on every chunk.",[426],"Re-embed only when the embedded text changed; update metadata otherwise.",[428],"Propagate deletes and permission changes explicitly, and test erasure end to end.",[430],"If you run CDC, alert on replication slot lag and retained WAL.",[432],"Keep a periodic sweep that catches what events missed.",[434],"Do model upgrades as blue\u002Fgreen migrations behind an alias, with rollback.",{"type":135,"content":436},[437,438,442,443,242],"If your RAG system needs to stay correct while the underlying data changes all day, see my ",{"tag":146,"to":439,"children":440},"\u002Fexpertise\u002Fai-engineer",[441],"AI engineering work",". The product-data side of the same problem is covered in ",{"tag":146,"to":444,"children":445},"\u002Fblog\u002Fpimcore-erp-delta-sync",[446],"the Pimcore ERP delta sync",{"type":152,"level":153,"id":131,"text":132},{"type":419,"ordered":205,"items":449},[450,453,456,459,462,465,468,471,474,477,480,483],[451],{"tag":170,"href":38,"children":452},[37],[454],{"tag":170,"href":41,"children":455},[40],[457],{"tag":170,"href":44,"children":458},[43],[460],{"tag":170,"href":47,"children":461},[46],[463],{"tag":170,"href":50,"children":464},[49],[466],{"tag":170,"href":53,"children":467},[52],[469],{"tag":170,"href":56,"children":470},[55],[472],{"tag":170,"href":59,"children":473},[58],[475],{"tag":170,"href":62,"children":476},[61],[478],{"tag":170,"href":65,"children":479},[64],[481],{"tag":170,"href":68,"children":482},[67],[484],{"tag":170,"href":71,"children":485},[70],[487,560,618,675],{"slug":488,"published":6,"minutes":489,"category":8,"tags":490,"keywords":495,"about":508,"sources":518,"cover":554,"og":555,"expertise":74,"locales":556,"lang":76,"title":557,"description":558,"coverAlt":559},"embedding-model-selection-2026",9,[491,10,492,493,494],"Embeddings","MTEB","Vector search","Model selection",[496,497,498,499,500,501,502,503,504,505,506,507],"best embedding model 2026","how to choose an embedding model","embedding model comparison","MTEB leaderboard explained","RTEB benchmark","open source embedding model","multilingual embedding model","voyage-4 vs openai embeddings","Qwen3-Embedding","EmbeddingGemma","embedding model for RAG","switching embedding models re-embed",[509,512,515],{"name":510,"url":511},"Word embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FWord_embedding",{"name":513,"url":514},"Information retrieval","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FInformation_retrieval",{"name":516,"url":517},"Discounted cumulative gain","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDiscounted_cumulative_gain",[519,522,525,528,531,534,537,539,542,545,548,551],{"title":520,"url":521},"Enevoldsen et al.: MMTEB, Massive Multilingual Text Embedding Benchmark (2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2502.13595",{"title":523,"url":524},"Hugging Face: Introducing RTEB, a new standard for retrieval evaluation (Oct 2025)","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Frteb",{"title":526,"url":527},"OpenAI: Vector embeddings guide","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fembeddings",{"title":529,"url":530},"OpenAI: API pricing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fpricing",{"title":532,"url":533},"Voyage AI: The Voyage 4 model family, shared embedding space with MoE architecture (Jan 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F01\u002F15\u002Fvoyage-4\u002F",{"title":535,"url":536},"Voyage AI: Text embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fembeddings",{"title":538,"url":71},"Voyage AI: Pricing",{"title":540,"url":541},"Google: Gemini API embeddings documentation","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fembeddings",{"title":543,"url":544},"Google: Gemini API pricing","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fpricing",{"title":546,"url":547},"Cohere: Embed models documentation","https:\u002F\u002Fdocs.cohere.com\u002Fdocs\u002Fcohere-embed",{"title":549,"url":550},"Qwen: Qwen3-Embedding-8B model card","https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-Embedding-8B",{"title":552,"url":553},"Google Developers Blog: Introducing EmbeddingGemma (Sep 2025)","https:\u002F\u002Fdevelopers.googleblog.com\u002Fen\u002Fintroducing-embeddinggemma\u002F","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fog.jpg",[76,77,78],"Choosing an embedding model in 2026: leaderboards, costs and a bake-off on your own data","How to pick an embedding model for RAG in 2026: what MTEB and RTEB tell you, hosted vs open weights, prices, dimensions, and a one-week bake-off.","Three columns of stacked blocks, the middle tallest, each measured by a short bar and a wide bracket.",{"slug":561,"published":6,"minutes":562,"category":8,"tags":563,"keywords":567,"about":579,"sources":587,"cover":612,"og":613,"expertise":74,"locales":614,"lang":76,"title":615,"description":616,"coverAlt":617},"contextual-retrieval-late-chunking",10,[10,564,491,565,566],"Chunking","Contextual retrieval","Late chunking",[568,569,570,571,572,573,574,575,576,577,578],"contextual retrieval","anthropic contextual retrieval","late chunking","contextual chunk embeddings","contextual embeddings rag","contextual bm25","chunk context rag","voyage-context-3","voyage-context-4","late chunking vs contextual retrieval","rag chunking context loss",[580,581,584],{"name":30,"url":31},{"name":582,"url":583},"Okapi BM25","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOkapi_BM25",{"name":585,"url":586},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",[588,591,594,597,600,603,606,609],{"title":589,"url":590},"Anthropic: Introducing Contextual Retrieval (Sep 2024)","https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fcontextual-retrieval",{"title":592,"url":593},"Günther et al.: Late Chunking, Contextual Chunk Embeddings Using Long-Context Embedding Models (2024, rev. 2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.04701",{"title":595,"url":596},"Jina AI: Late Chunking in Long-Context Embedding Models (Aug 2024)","https:\u002F\u002Fjina.ai\u002Fnews\u002Flate-chunking-in-long-context-embedding-models\u002F",{"title":598,"url":599},"Voyage AI: Introducing voyage-context-3 (Jul 2025)","https:\u002F\u002Fblog.voyageai.com\u002F2025\u002F07\u002F23\u002Fvoyage-context-3\u002F",{"title":601,"url":602},"Voyage AI: voyage-context-4 (Jun 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F06\u002F29\u002Fvoyage-context-4\u002F",{"title":604,"url":605},"Voyage AI: Contextualized chunk embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fcontextualized-chunk-embeddings",{"title":607,"url":608},"Anthropic: Claude API pricing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fpricing",{"title":610,"url":611},"Qu, Tu and Bao: Is Semantic Chunking Worth the Computational Cost? (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13070","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fog.jpg",[76,77,78],"Contextual retrieval and late chunking: giving RAG chunks back their context","Chunks lose their document. How contextual retrieval, late chunking and contextualized embedding models fix that, what each costs, and when to skip it.","Four loose square pieces gathered by a wide bracket below, each joined upward to a large sheet.",{"slug":619,"published":6,"minutes":489,"category":8,"tags":620,"keywords":624,"about":637,"sources":645,"cover":669,"og":670,"expertise":74,"locales":671,"lang":76,"title":672,"description":673,"coverAlt":674},"rag-access-control-permissions",[10,621,622,623,493],"Access control","Security","Multi-tenancy",[625,626,627,628,629,630,631,632,633,634,635,636],"rag access control","permission aware rag","document level security rag","rag multi tenancy","vector database access control","pgvector row level security","security trimming azure ai search","openfga rag authorization","qdrant multitenancy","pinecone namespaces multitenancy","owasp llm08 vector and embedding weaknesses","rag data leakage",[638,641,644],{"name":639,"url":640},"Access-control list","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FAccess-control_list",{"name":642,"url":643},"Multitenancy","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMultitenancy",{"name":30,"url":31},[646,649,652,654,657,660,663,666],{"title":647,"url":648},"OWASP GenAI Security Project: LLM08:2025 Vector and Embedding Weaknesses","https:\u002F\u002Fgenai.owasp.org\u002Fllmrisk\u002Fllm082025-vector-and-embedding-weaknesses\u002F",{"title":650,"url":651},"PostgreSQL documentation: Row Security Policies","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fddl-rowsecurity.html",{"title":653,"url":62},"pgvector README: filtering and iterative index scans",{"title":655,"url":656},"Qdrant: Multitenancy and partitions guide","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fguides\u002Fmultiple-partitions\u002F",{"title":658,"url":659},"Pinecone: Implement multitenancy using namespaces","https:\u002F\u002Fdocs.pinecone.io\u002Fguides\u002Findex-data\u002Fimplement-multitenancy",{"title":661,"url":662},"Microsoft Learn: Document-level access control in Azure AI Search","https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fsearch-document-level-access-overview",{"title":664,"url":665},"OpenFGA: RAG authorization use case","https:\u002F\u002Fopenfga.dev\u002Fdocs\u002Fuse-cases\u002Frag-authorization",{"title":667,"url":668},"Morris et al.: Text Embeddings Reveal (Almost) As Much As Text (EMNLP 2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.06816","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fog.jpg",[76,77,78],"Access control for RAG: permission-aware retrieval without leaking documents","How RAG assistants leak documents and how to stop it: ACLs on every chunk, pre-filtering, row-level security in pgvector, permission sync and leak tests.","Seven cards in a row, six with a lock badge, one bare card sending an arrow outward.",{"slug":676,"published":6,"minutes":562,"category":8,"tags":677,"keywords":682,"about":694,"sources":702,"cover":729,"og":730,"expertise":731,"locales":732,"lang":76,"title":733,"description":734,"coverAlt":735},"rag-product-catalog-pim-data",[10,678,679,680,681],"PIM","Product data","B2B e-commerce","Metadata filtering",[683,684,685,686,687,688,689,690,691,692,693],"rag product catalog","rag pim data","llm product data","product catalog chatbot","akeneo rag","pimcore ai search","rag structured data","metadata filtering rag","product attributes embeddings","b2b product assistant","etim classification ai",[695,698,699],{"name":696,"url":697},"Product information management","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FProduct_information_management",{"name":30,"url":31},{"name":700,"url":701},"ETIM (standard)","https:\u002F\u002Fwww.etim-international.com\u002Fclassification\u002F",[703,706,709,712,714,717,720,723,726],{"title":704,"url":705},"Akeneo API: Products, product values, metric values and product models","https:\u002F\u002Fapi.akeneo.com\u002Fconcepts\u002Fproducts.html",{"title":707,"url":708},"Akeneo Help Center: What about products with variants?","https:\u002F\u002Fhelp.akeneo.com\u002Fserenity-take-the-power-over-your-products\u002Fserenity-what-about-products-with-variants",{"title":710,"url":711},"Pimcore documentation: Classification Store","https:\u002F\u002Fdocs.pimcore.com\u002Fplatform\u002FPimcore\u002FObjects\u002FObject_Classes\u002FData_Types\u002FClassification_Store\u002F",{"title":713,"url":701},"ETIM International: Classification",{"title":715,"url":716},"ETIM International: ETIM releases and the IXF format","https:\u002F\u002Fetim-international.com\u002F?p=754",{"title":718,"url":719},"Reddy et al.: Shopping Queries Dataset, a large-scale ESCI benchmark for improving product search (2022)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2206.06588",{"title":721,"url":722},"Amazon Science: esci-data repository","https:\u002F\u002Fgithub.com\u002Famazon-science\u002Fesci-data",{"title":724,"url":725},"Qdrant documentation: Filtering","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fconcepts\u002Ffiltering\u002F",{"title":727,"url":728},"Sui et al.: Table Meets LLM, can large language models understand structured table data? (WSDM 2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.13062","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fog.jpg","b2b-ecommerce-developer",[76,77,78],"RAG over product catalogues: turning PIM data into answers that hold","Product data is structured, so chunking it like PDFs fails. How to serialize PIM attributes, route spec questions to filters and keep answers exact.","A frame of nine squares with tiny symbols, funnelling down to one card.",1792342719226]