Web content ingestion for RAG: what to keep after the fetch

Ingestion is the step between "we fetched the page" and "we chunked it". It is easy to rush, because nothing visibly breaks when it is wrong. The damage shows up later: a retriever that cannot tell two versions of a page apart, a citation that points at a redirect, an index that quietly lost a fifth of its sources. This guide covers what to keep from each page so the later steps have something to work with.

Why ingestion decides retrieval quality

Everything downstream reads from what ingestion stored. The chunker can only keep a heading path if the headings survived. The retriever can only filter by source or date if those fields exist. The answer can only cite a page correctly if you stored the URL that the page actually lives at. And a refresh job can only re-embed what changed if you kept something to compare against.

Web pages make this easier than most document types, because they arrive with structure: a title, headings, a URL, often a date. The mistake is to throw that structure away at the start and pay a model to reconstruct it later.

Start from a source list, not a crawl

Keep the pages you index in a reviewed list, under version control, grouped by source type. Three reasons:

• Focus. Every page in the list is one you would be willing to quote. A crawl indexes whatever links lead to, including tag pages, archives and duplicates.
• Cost. Fetch and embedding costs grow with the number of pages and how often you refresh them. A list makes both numbers explicit.
• Accountability. When someone asks why the assistant said something, you can show where the page came from and why it was included.

Stay with public pages. Pages behind a login, a paywall or other access control are out of scope. Read each site's terms of use and its robots.txt, which is specified in RFC 9309.¹ Neither replaces the other, and none of this is legal advice: if your use case raises copyright or data-protection questions, ask counsel.

Clean Markdown, and what main content leaves out

Fetch each page as Markdown with the main content extracted, so navigation, footers and cookie banners are gone before you embed anything. Markdown is the right intermediate format for retrieval because it keeps what matters for chunking: headings, lists, tables and code blocks.

Main-content extraction is a heuristic, so check its output once per source type:

• Residue. "Related articles", "Was this page helpful?" and share prompts sometimes survive. Strip them with a short per-source rule list.
• Over-trimming. Some layouts put real content in sidebars, such as version notes or parameter tables. If you need it, fetch that source with the full body and clean it yourself.
• Empty pages. A page that comes back with a title and almost no text was probably assembled in the browser by JavaScript. Record it as a failure and find another source for the same information, such as a static docs page or a published PDF.

A metadata schema you can copy

Store these fields on every page and copy the page-level ones onto every chunk:

Two details are easy to get wrong. First, published_date and retrieved_at answer different questions ("how old is the claim?" and "how old is our copy?"), so never fill one from the other. Second, if your fetcher serves from a cache, retrieved_at is when you received the copy, not when the page was last read from the origin. Store the cache status next to it so you can tell the two apart.

IDs, canonical URLs and duplicates

Use the final URL as the document ID and derive chunk IDs from it, for example https://example.com/docs/a#3. Normalise it once: lowercase the host, drop tracking parameters such as utm_*, drop the fragment, and keep the query string only when it changes the content.

Sites often expose the same content under several URLs: with and without a trailing slash, print views, old paths that redirect. Google's canonicalization guide describes how sites signal the preferred version.² For your index, the practical rule is simpler: collapse URLs that redirect to the same final URL, and catch the rest by comparing content hashes. Two pages with the same hash are one document with two addresses; keep one and record the other as an alias.

Near-duplicates, such as the same article with a different date line, need a fuzzier check. Shingling or MinHash over the cleaned text works well enough for a few thousand pages. Deduplicate before embedding, because duplicates crowd out other results in the top-k.

Frameworks can track some of this for you. LlamaIndex's ingestion pipeline keeps a map of document IDs to content hashes in a docstore, skips unchanged documents and upserts changed ones.³ LangChain's indexing API does similar bookkeeping with a record manager.⁴ Both work better when the ID you give them is stable, which is another reason to use the final URL.

Failed and empty pages: record them, do not skip them

A batch job that logs "98 of 120 pages indexed" and moves on has lost information you will need. Keep a failure log with the URL, the error, the time and the run, and decide per error what to do:

Report three numbers per run: pages fetched, pages failed and pages changed. A jump in failures for one source usually means the site changed its structure or moved its content.

Where the fetch step fits

Everything above works with any fetcher. If you use Live Web Data API for the fetch, the response already carries several of these fields: results[].url is the final URL after redirects, title, author and publishedDate are returned when the page exposes them, statuses[].fetchedAt says when the page was read (every request is live; there is no cache), and each requested URL gets its own status with an error tag such as CRAWL_NOT_FOUND or CRAWL_NOT_RENDERED. Failed URLs cost 0 credits, so recording a failure is free. The API fetches one or more URLs per request; it does not crawl, chunk, embed or store. Markdown is truncated at text.maxCharacters, 100,000 characters by default: raise it for any page that hits the limit before chunking or hashing, and record every page with results[].textTruncated: true as truncated, so a cut-off copy never passes as complete. The automation docs cover batch fetching and retries, the SDKs wrap the request, and the API reference lists the per-request URL limit and every per-URL error tag.

Where to go next

With clean Markdown and a metadata schema in place, the next step is splitting pages into chunks that keep their context: read chunking Markdown for RAG. To decide how often to re-fetch each source group, see RAG index freshness.

Request API access. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

Footnotes

1. RFC 9309, "Robots Exclusion Protocol" (2022). ↩

2. Google Search Central, "What is URL canonicalization". ↩

3. LlamaIndex documentation, "Ingestion pipeline", document management with a docstore. ↩

4. LangChain reference, langchain_core.indexing API. ↩