Skip to content

Freshness

Delete stale vectors without deleting pages that only failed once

Removed pages keep answering until their vectors go. Delete on a confirmed 404 or 410, never on a timeout, and watch sync modes that delete what is missing.

Published

This article was drafted with AI assistance from the sources listed below and edited and published by Teneo Labs. Disclosure: this site is run by Teneo Labs, the company behind Live Web Data API, which this article mentions.

Before a refresh job deletes anything
  • Every chunk carries its page ID, so one page can be deleted in one call
  • 410 deletes at once; 404 deletes after it repeats on the next run
  • Timeouts and temporary errors keep the previous version
  • Failed pages are passed to the sync step as unchanged, not left out
  • A redirect re-keys the page to its new final URL
  • The run logs how many pages it deleted, per source group

A page that was removed from the web keeps answering questions until its vectors are deleted. The fix sounds simple: delete what is gone. The trap is that "gone" and "failed to load this time" look the same to a careless refresh job.

Gone is a status code, not a missing result

RFC 9110 defines 404 Not Found as "no current representation" and 410 Gone as a removal that is likely permanent and intentional.1 A 410 is safe to act on at once. A 404 can be a temporary mistake on the site, so delete after it repeats on the next run.

Everything else, such as timeouts, server errors, rate limits and empty responses, is a failure to read, not evidence of removal. Keep the previous version and retry.

Watch out for sync modes

Frameworks make deletion easy, sometimes too easy. LangChain's indexing API has cleanup modes: incremental removes old versions of documents you re-index, while full also deletes every document that is not in the batch you pass in.2 LlamaIndex's ingestion pipeline with a docstore offers an upserts-and-delete strategy that removes documents missing from the input.3

Both behave correctly for what they are given. The risk is upstream: if your job drops pages that failed to fetch from the batch, a full sync deletes them. One bad network minute can empty part of an index.

Two safe patterns:

  • Pass failures through. For pages that failed this run, pass the previous version to the sync step, so they count as unchanged.
  • Delete explicitly. Use an incremental mode for updates, and delete by page ID only for pages your own rule marks as gone.

Delete the whole page at once

Give every chunk the page ID, the final URL after redirects, so one delete call removes all of a page's chunks. Deleting chunk by chunk leaves fragments that keep matching queries. The index freshness guide covers the related case of replacing a changed page in one step.

With Live Web Data API

Each requested URL gets its own status. A CRAWL_NOT_FOUND error marks a page that returned 404 or 410, while timeouts and other errors carry their own tags, so the rule above maps directly onto the response. Every request is a live fetch with no cache, so the tag reflects the page as it is now; still confirm a removal on a second run before deleting, because a site can return a 404 briefly during a deploy. The API reference lists every per-URL error tag. The API fetches; deciding what to delete and deleting it stay in your pipeline.

Request API access. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

Footnotes

  1. RFC 9110, "HTTP Semantics", sections 15.5.5 and 15.5.11. ↩

  2. LangChain reference, langchain_core.indexing API, cleanup modes. ↩

  3. LlamaIndex documentation, "Ingestion pipeline", document management. ↩

Frequently asked questions

Why do deleted web pages still show up in RAG answers?

Because nothing removed their chunks. A vector store only changes when your pipeline writes to it. Unless the refresh job deletes a page's chunks when the page is gone, retrieval keeps returning them.

Should a timeout remove a page from a RAG index?

No. A timeout says nothing about whether the page still exists. Keep the previous version, retry on the next run, and delete only on a confirmed 404 or 410.

Sources

  1. langchain_core.indexing API reference (Documentation)
  2. Ingestion pipeline: document management (LlamaIndex docs) (Documentation)
  3. RFC 9110: HTTP Semantics (404 and 410 status codes) (Documentation)