> Keeping a RAG index current: staleness budgets, dirty tracking, CDC from Postgres with Debezium, deletes and permissions, blue/green swaps and costs.
>
> Web page: https://balazscsorba.com/blog/incremental-indexing-cdc · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/incremental-indexing-cdc.md) · [Magyar](https://balazscsorba.com/hu/blog/incremental-indexing-cdc.md)
> Author: Balázs Csorba · Published: 2026-09-25 · Updated: 2026-10-17 · Keywords: incremental indexing for RAG, change data capture RAG, Debezium Postgres logical decoding, re-embedding cost, blue/green index swap vector database, index staleness monitoring, delete propagation vector index, permission changes RAG index, Qdrant aliases, high water mark change detection

[Blog](https://balazscsorba.com/blog)/Retrieval & search

# Keeping a RAG index fresh: incremental indexing, CDC and blue/green re-embedding

Keeping a RAG index current: staleness budgets, dirty tracking, CDC from Postgres with Debezium, deletes and permissions, blue/green swaps and costs.

[Balázs Csorba](https://balazscsorba.com/about)·September 25, 2026·updated October 17, 2026·8 min read

-   RAG
-   Change data capture
-   PostgreSQL
-   Vector databases
-   Data freshness

![A database streams changes into a cabinet, with a second cabinet and a switch beside it.](https://balazscsorba.com/images/blog/incremental-indexing-cdc/cover.webp?v=5761ce5458)

## Key takeaways

-   Give every source a staleness budget in minutes and measure lag against it; “eventually consistent” is not a requirement anyone can test.
-   Watermarks and content hashes are enough for many sources; CDC from Postgres via logical decoding and Debezium is for sources where minutes matter, and its replication slots must be monitored.
-   Deletes and permission changes are the deltas indexers miss: Azure AI Search documents that indexers do not track deletions, and GDPR erasure must happen without undue delay.
-   Re-embedding for a model upgrade belongs in a new collection behind an alias, switched with one API call and kept for rollback; in pgvector, build indexes concurrently.
-   The tokens are the cheap part: re-embedding 100 million tokens costs about $2 at $0.02 per million; engineering, backfill time and correctness are the real cost.

On this page

1.  [What fresh means: a staleness budget](https://balazscsorba.com/#staleness-budget)
2.  [Dirty tracking with watermarks and hashes](https://balazscsorba.com/#dirty-tracking)
3.  [Change data capture from Postgres](https://balazscsorba.com/#cdc-from-postgres)
4.  [Deletes and permission changes](https://balazscsorba.com/#deletes-and-permissions)
5.  [Full reindex and blue/green swaps](https://balazscsorba.com/#reindex-and-blue-green-swaps)
6.  [The cost of re-embedding](https://balazscsorba.com/#cost-of-re-embedding)
7.  [When CDC is overkill](https://balazscsorba.com/#when-not-to-use-cdc)
8.  [A checklist for an index that stays current](https://balazscsorba.com/#checklist)
9.  [Sources](https://balazscsorba.com/#sources)

Listen to this article

0:000:00

**Incremental indexing for RAG** is the layer that keeps the index true after the first crawl: new documents get chunked and embedded, changed ones get replaced, deleted ones disappear, and permission changes reach the index before someone sees a document they should not. Most RAG prototypes skip it because the first demo uses a static folder. Production corpora change all day, and an assistant that cites last week’s price list or a revoked policy is worse than one that says it does not know.

This article covers a staleness budget, dirty tracking with watermarks and hashes, change data capture from Postgres, deletes and permission changes, full reindexing with blue/green swaps, the cost of re-embedding and when CDC is overkill. Chunking, retrieval and reranking are covered in [RAG pipelines: chunking, hybrid search and reranking](https://balazscsorba.com/blog/rag-pipeline-chunking-hybrid-search-reranking); versions and prices are as of October 2026.

## What fresh means: a staleness budget

Fresh means a measured number per source, not a feeling. Write down, for each source, how many minutes a change may take to reach the index: perhaps five for stock levels, an hour for product descriptions, a day for archived manuals, and immediately for revoked access. No standard prescribes these numbers; they come from what a wrong answer costs. Then measure the actual lag from source commit to index write, and alert when it exceeds the budget.

The budget also tells you which mechanism you need. A daily budget is met by a nightly sweep; a five-minute budget needs either a frequent scan or events; a budget of seconds for permissions usually means checking access at query time as well as in the index. Splitting the index helps here. In Spryker and Pimcore catalogs, price and stock change all day while long descriptions change rarely, so volatile scalar fields can be updated as metadata without re-embedding the text they sit next to.

## Dirty tracking with watermarks and hashes

Most sources do not need a database log, only a reliable way to find what changed since the last run. A high-water-mark policy is the classic pattern: remember the largest `updated_at` value processed and fetch rows above it next time. Azure AI Search names it as a reusable change-detection policy, and its scheduled indexers run between every 5 minutes and every 24 hours ([Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-howto-schedule-indexers)). Use a monotonic column, keep a small overlap window and make writes idempotent, because clocks and long transactions produce rows that commit with an older timestamp.

Content hashes complement watermarks. Hash the text that will be embedded, not the whole row, and skip re-embedding when the hash is unchanged; a price update then touches metadata only. The same idea scales down to files and repositories, where a hash per file decides what to re-chunk, as described in [RAG over code](https://balazscsorba.com/blog/code-rag-repository-indexing). The weak spot of both is deletion: a row that no longer exists never shows up in a query for changed rows.

A sweep also needs a backstop for drift. Once a week, compare the full list of source ids with the ids present in the index: anything in the index but not in the source is a missed delete, and anything in the source but not in the index is a missed insert. That comparison is cheap, it catches bugs in every other mechanism, and its result is a number you can put on the same dashboard as the lag.

## Change data capture from Postgres

Change data capture reads committed changes from the database’s write-ahead log instead of polling tables, so inserts, updates and deletes all arrive as events within seconds. In Postgres the mechanism is logical decoding, “the process of extracting all persistent changes to a database’s tables into a coherent, easy to understand format”, which relies on replication slots and an output plugin ([PostgreSQL](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)). A publication defines which tables are in scope ([CREATE PUBLICATION](https://www.postgresql.org/docs/current/sql-createpublication.html)), but schema changes are not replicated, so DDL is a separate job ([restrictions](https://www.postgresql.org/docs/current/logical-replication-restrictions.html)).

![Event lane for a fresh index Five steps from left to right: a change is committed in Postgres; logical decoding exposes it through a replication slot; Debezium turns it into an insert, update or delete event; a transform filters relevant fields, re-chunks and embeds changed text; and the index upserts or deletes the affected chunks. A scheduled sweep by updated_at catches anything the event lane missed, and a lag alarm watches the replication slot.](https://balazscsorba.com/images/blog/incremental-indexing-cdc/diagram-11.webp?v=00ff24f199)

The event lane delivers changes in seconds; a slower sweep and a lag alarm make sure nothing stays wrong for long.

Debezium is the common way to consume it. Its PostgreSQL connector needs `wal_level=logical` and enough WAL senders and replication slots, and its documentation is blunt about the risk: replication slots retain all WAL that Debezium needs even during outages, so they must be monitored to avoid disk exhaustion and catalog bloat ([Debezium](https://debezium.io/documentation/reference/stable/connectors/postgresql.html)). Update events include the previous column values only with `REPLICA IDENTITY FULL`; a heartbeat interval keeps the slot advancing when the captured tables are quiet, and a signaling table triggers targeted backfills instead of a full reindex.

```
-- Postgres side of CDC: scope, slot health and lag
ALTER TABLE products REPLICA IDENTITY FULL;            -- update events carry old values too
CREATE PUBLICATION rag_sources FOR TABLE products, product_texts, document_acl;

-- WAL retained by each slot: alert long before the disk fills
SELECT slot_name, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS behind
FROM pg_replication_slots;
```

Postgres exposes the numbers you need for the lag alarm: `pg_replication_slots` shows each slot’s confirmed position and WAL status, and `max_slot_wal_keep_size` caps how much WAL a slot may hold before it is invalidated ([PostgreSQL](https://www.postgresql.org/docs/current/view-pg-replication-slots.html)). Alert on the gap between the current WAL position and the slot’s confirmed position long before it reaches that cap. For order and catalog data that already flows between systems, the same outbox and idempotency patterns apply, see [order sync between shop and ERP](https://balazscsorba.com/blog/b2b-order-sync-erp).

## Deletes and permission changes

Deletes and permission changes are the two deltas that indexers most often miss, and they are the two with legal consequences. Azure AI Search states it plainly: an indexer does not track object deletion in data sources, so a soft-delete strategy is needed, and the policy must be applied from the very first indexer run or deleted documents stay in the index ([Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-howto-index-changed-deleted-blobs)). Under GDPR Article 17 personal data must be erased without undue delay once erasure is due ([GDPR](https://gdpr-info.eu/art-17-gdpr/)), and a vector index holding chunks of that data is part of the scope.

Handle both explicitly. Store the source id and document id on every chunk so a delete can remove all chunks of a document in one call; consume delete events from CDC where you have them; and run a periodic reconciliation that removes chunks whose source no longer exists. For access control, keep the permission state in the index next to the text and vectors, and propagate permission changes as their own events. Azure AI Search now stores ACL metadata inside the index and added per-run incremental ACL sync for its SharePoint indexer in 2026 ([Azure AI Search](https://learn.microsoft.com/en-us/azure/search/whats-new)).

**A deleted document is still answerable until you delete its chunks**

Removing a document from the source system does not remove it from the vector index, the cache in front of it or any answer log. Test erasure end to end: delete a known document, then ask the assistant a question only that document could answer, and check that the citation is gone.

## Full reindex and blue/green swaps

Some changes cannot be applied incrementally: a new embedding model, a new chunking strategy or a new index configuration means every vector changes. Treat that as a migration with a blue/green swap. Qdrant documents aliases for exactly this case, switching between versions of vectors “when upgrading to a new version of the neural network”, with alias changes sent as one actions request ([Qdrant](https://qdrant.tech/documentation/concepts/collections/)); Elasticsearch aliases serve the same purpose and are documented as a way to reindex without downtime ([Elastic](https://www.elastic.co/docs/manage-data/data-store/aliases)).

![Blue/green re-embedding timeline Four phases from left to right on one timeline. Build: a new collection v2 is created with the new model while v1 serves all traffic. Backfill: v2 is filled from the source and new changes are written to both. Switch: one alias call points the application from v1 to v2. Keep: v1 stays read-only for rollback, then it is dropped. Below, the pgvector variant: create the new index concurrently, then switch in application configuration.](https://balazscsorba.com/images/blog/incremental-indexing-cdc/diagram-21.webp?v=438e1a5486)

A model upgrade is a migration: build beside, backfill, switch with one call, and keep the old index until the new one has proven itself.

The sequence is the same everywhere: build the new collection with the new model while the old one serves, backfill it from the source, write new changes to both during the backfill, switch the alias, and keep the old collection read-only until the new one has passed your retrieval evaluation in production. The documentation describes the alias call, not its transactional guarantees under load, so test the switch on a staging copy first. In pgvector, the equivalent is a new column or table, an index built with `CREATE INDEX CONCURRENTLY` so writes are not blocked, and a switch in application configuration ([pgvector](https://github.com/pgvector/pgvector)).

pgvector adds two practical notes: index builds are much faster when the HNSW graph fits into `maintenance_work_mem`, and the defaults of `m = 16` and `ef_construction = 64` are a reasonable start for the new index. Choosing the model you migrate to is a separate decision, covered in [embedding model selection](https://balazscsorba.com/blog/embedding-model-selection-2026).

## The cost of re-embedding

The tokens are the cheap part. As of October 2026, OpenAI lists `text-embedding-3-small` at $0.02 and `text-embedding-3-large` at $0.13 per million tokens ([OpenAI](https://platform.openai.com/docs/pricing)), and Voyage lists `voyage-4-lite` at $0.02 and `voyage-4` at $0.06 per million, with a batch discount ([Voyage](https://docs.voyageai.com/docs/pricing)). Re-embedding a corpus of 100 million tokens therefore costs roughly $2 to $13 in API fees. That is arithmetic on list prices, not a quote, and it is small next to everything else.

The real costs are elsewhere: hours of backfill under rate limits, a dual-write window that must be correct, extra storage while two indexes exist, and a retrieval evaluation on the new model before the switch. On my own Nuxt site, where I run LLM features over a small corpus, re-embedding everything is trivial, and that is exactly why the token bill is not a reason to build CDC; freshness, deletes and permissions are.

One shortcut is worth checking before a full re-embed. Some vendors describe model families whose members share an embedding space, which would allow upgrading the query side first; verify such a claim in the vendor’s own documentation and with your evaluation set before relying on it, because mixing vectors from incompatible models silently degrades retrieval.

## When CDC is overkill

CDC is overkill when the staleness budget is hours or a day, when the source is a SaaS system without database access, or when nobody on the team will watch replication slots. A nightly or hourly sweep with watermarks, content hashes and a deletion reconciliation is simpler, easier to reason about and enough for most document collections. For SaaS sources, webhooks are the event path: verify a secret, respond within 10 seconds, check the event type, redeliver missed deliveries and deduplicate on the delivery id, as GitHub’s guidance describes ([GitHub](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks)).

Mechanism

Typical staleness

Catches deletes

Catches permission changes

Operational load

Nightly full reindex

up to a day

yes, by rebuild

yes, by rebuild

low, but costly at scale

updated\_at sweep

minutes to hours

no, needs reconciliation

only if ACL rows have timestamps

low

Content-hash diff

per run

yes, if the full list is compared

no

low

CDC via logical decoding

seconds

yes, delete events

yes, if ACL tables are published

high: slots, lag, schema changes

Inbound webhooks

seconds

if the source sends them

if the source sends them

medium: retries, dedupe

## A checklist for an index that stays current

1.  Set a staleness budget per source and measure lag against it.
2.  Store source id, document id, content hash and permissions on every chunk.
3.  Re-embed only when the embedded text changed; update metadata otherwise.
4.  Propagate deletes and permission changes explicitly, and test erasure end to end.
5.  If you run CDC, alert on replication slot lag and retained WAL.
6.  Keep a periodic sweep that catches what events missed.
7.  Do model upgrades as blue/green migrations behind an alias, with rollback.

If your RAG system needs to stay correct while the underlying data changes all day, see my [AI engineering work](https://balazscsorba.com/expertise/ai-engineer). The product-data side of the same problem is covered in [the Pimcore ERP delta sync](https://balazscsorba.com/blog/pimcore-erp-delta-sync).

## Sources

1.  [PostgreSQL docs: logical decoding concepts](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)
2.  [Debezium docs: PostgreSQL connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html)
3.  [PostgreSQL docs: CREATE PUBLICATION](https://www.postgresql.org/docs/current/sql-createpublication.html)
4.  [PostgreSQL docs: pg\_replication\_slots](https://www.postgresql.org/docs/current/view-pg-replication-slots.html)
5.  [Azure AI Search: changed and deleted blobs](https://learn.microsoft.com/en-us/azure/search/search-howto-index-changed-deleted-blobs)
6.  [Azure AI Search: scheduling indexers](https://learn.microsoft.com/en-us/azure/search/search-howto-schedule-indexers)
7.  [Qdrant docs: collections and aliases](https://qdrant.tech/documentation/concepts/collections/)
8.  [Elastic docs: aliases](https://www.elastic.co/docs/manage-data/data-store/aliases)
9.  [pgvector on GitHub](https://github.com/pgvector/pgvector)
10.  [GDPR Article 17](https://gdpr-info.eu/art-17-gdpr/)
11.  [OpenAI API pricing](https://platform.openai.com/docs/pricing)
12.  [Voyage AI docs: pricing](https://docs.voyageai.com/docs/pricing)

## Frequently asked questions

How do I keep a RAG index up to date?

Track which source records changed, by watermark, content hash or change data capture, re-chunk and re-embed only those, propagate deletes and permission changes explicitly, and run a periodic sweep that catches anything the event path missed. Measure the lag against a staleness budget per source.

What is change data capture for RAG?

Reading changes from the database’s log instead of polling tables. In Postgres, logical decoding exposes committed inserts, updates and deletes, and Debezium turns them into events that an indexer can consume within seconds. It needs replication slots, which retain WAL until consumed and must be monitored.

How do deletes reach the vector index?

Only if you send them. Many indexers do not track deletion in the source, so use delete events from CDC, a soft-delete flag the indexer reads, or a periodic reconciliation that removes chunks whose source id no longer exists. Personal data under GDPR must be erased without undue delay.

How do I re-embed with a new model without downtime?

Build a new collection with the new model while the old one serves, backfill it and dual-write new changes to both, then switch an alias to the new collection in one call and keep the old one read-only for rollback. In pgvector, create the new index concurrently and switch in configuration.

How much does re-embedding a corpus cost?

At embedding API prices of $0.02 to $0.13 per million tokens as of October 2026, re-embedding 100 million tokens costs roughly $2 to $13. The real costs are the backfill time, rate limits, the engineering of the swap, and re-checking retrieval quality on the new model.

Written by Balázs Csorba

Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents.

[AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about)

## More articles

-   [Choosing an embedding model in 2026: leaderboards, costs and a bake-off on your own data](https://balazscsorba.com/blog/embedding-model-selection-2026)
-   [Contextual retrieval and late chunking: giving RAG chunks back their context](https://balazscsorba.com/blog/contextual-retrieval-late-chunking)
-   [Access control for RAG: permission-aware retrieval without leaking documents](https://balazscsorba.com/blog/rag-access-control-permissions)
-   [RAG over product catalogues: turning PIM data into answers that hold](https://balazscsorba.com/blog/rag-product-catalog-pim-data)

## Sounds like what you need?

Tell me about your project or role – I’d love to hear from you.

[Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)