RAG index freshness: re-fetch, re-embed, delete what is gone

A web-sourced index starts going stale the day you build it. Pages get edited, moved and retired, and nothing in the vector store notices. The result is the write-once index: confident answers from a version of the web that no longer exists. This guide covers how to keep an index true: how often to refresh, how to detect real changes, how to update without mixing versions, and when to delete.

How an index goes stale

There are three different kinds of staleness, and each needs its own fix:

• Edited pages. The URL is the same, the content is not. Answers quote text that is no longer there.
• Removed pages. The URL now returns 404 or 410, but its chunks are still retrieved. Retired features and withdrawn policies keep being explained.
• Moved pages. The old URL redirects to a new one. Without a check, you end up with two copies, or with citations that point at a redirect.

A refresh job that only re-fetches and re-embeds handles the first, partially. Handling all three needs IDs, hashes and an explicit deletion step.

Cadence per source type

Refreshing everything daily wastes fetches and embeddings on pages that change twice a year. Refreshing everything monthly leaves fast-moving pages wrong for weeks. Group sources by how they change and give each group its own interval:

These are starting points, not measurements. After a few runs, look at how many pages per group actually changed. If a weekly group shows almost no changes, slow it down; if a monthly group changes every run, speed it up. Pages that are cited often deserve a shorter interval than their group.

Change detection with content hashes

Compare a hash of the cleaned Markdown, not of the raw HTML. Raw HTML changes on every request on many sites, because of rotating ads, session tokens or build IDs. Hashing the extracted main content means only real text changes trigger re-embedding.

Normalise before hashing: collapse whitespace, and strip lines you already treat as noise, such as "Last updated" stamps that change without the content changing. Store the hash on the page record and on every chunk.

LlamaIndex's ingestion pipeline does this bookkeeping when you attach a docstore: it maps each document ID to a hash, skips unchanged documents and upserts changed ones.¹ LangChain's indexing API does the same with a record manager and offers cleanup modes that remove old versions of changed documents.² Both depend on a stable document ID; use the final URL after redirects.

Incremental upserts and deletions

When a page changes, replace all of its chunks at once:

1. Re-chunk the new version with the current chunker.
2. Embed the new chunks.
3. Write the new chunks and delete every old chunk with that page ID in the same transaction, or behind an index alias you switch atomically.

Updating chunk by chunk leaves windows where old and new sections are retrieved together, which produces answers that mix two versions of a page.

Deletions need more care than updates, because a failed fetch is not the same as a removed page:

RFC 9110 defines 410 Gone as an intentional, likely permanent removal, so it is safe to act on immediately.³ A 404 can be a temporary mistake on the site's side, so confirming it on a second run is a reasonable default.

Fetch age in refresh jobs

If your fetcher serves cached copies, a refresh run can quietly re-embed an old version. Know which kind of fetch you get. If it caches, set the maximum cache age to match the group's interval; if it reads live, keep the last fetch time per URL in your own store and skip URLs whose copy is still younger than the group's interval, so two runs in the same window do not fetch the same page twice.

Live Web Data API reads every page live and keeps no cache, so each run sees the current version. Store statuses[].fetchedAt with every page and compare it with the group's interval before the next run. A CRAWL_NOT_FOUND error marks a page that returned 404 or 410 when it was read live, so it can feed the deletion rule above directly; still confirm a 404 on the next run. The API cuts content at text.maxCharacters, 100,000 characters by default, and sets results[].textTruncated when it did; raise the limit for any page that hits it before hashing, or an edit past the cut never changes the hash. The automation docs cover batching several URLs per request and retries, and the SDKs wrap the request. The API fetches; hashing, upserts and deletes stay in your pipeline.

Stale citations: re-check before you answer

Even with a good schedule, a page can change the day after a refresh. For answers that people act on, re-fetch the cited pages live before showing the answer, or at least show the date each source was read. RAG citations from web sources covers the re-check in detail.

What to log

Every run should report, per source group: pages fetched, pages changed, pages failed (by error), pages deleted and the oldest retrieved_at still in the index. The last number is the one to alert on. It is the honest answer to "how out of date can this assistant be?"

Where to go next

Freshness depends on decisions made at ingestion: stable IDs, hashes and a failure log. If those are missing, start with web content ingestion for RAG.

Request API access. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

Footnotes

1. LlamaIndex documentation, "Ingestion pipeline", document management with a docstore. ↩

2. LangChain reference, langchain_core.indexing API (record manager and cleanup modes). ↩

3. RFC 9110, "HTTP Semantics", sections 15.5.5 (404 Not Found) and 15.5.11 (410 Gone). ↩