RAG retrieval for web pages: evaluate, hybrid search, rerank

Retrieval is where most RAG answers are won or lost. If the passage with the answer is not in what the model sees, no prompt will fix it. The tempting move is to add techniques: a reranker, query rewriting, a graph. The better move is to measure first, so you know which problem you have. This guide covers a small evaluation set, then the techniques in the order they usually pay off for web-sourced indexes.

Measure first: a golden question set

Before tuning anything, write down 30 to 50 questions that real users ask or will ask. For each one, record the URL and section that contains the answer. This takes an afternoon and is the most useful asset in the whole pipeline.

Then measure two things:

• Recall at k. For what share of questions does a correct chunk appear in the top k results? This tells you whether retrieval can work at all.
• Rank of the first correct chunk. Mean reciprocal rank (MRR) summarises it. It tells you whether the right chunk is near the top or buried.

Keep the set small enough that you read every miss. A miss is either a retrieval miss (the right chunk was not returned) or an ingestion miss (the right chunk does not exist, for example because a table was split or a page failed to load). The fixes are different, and only reading the misses tells you which you have. Frameworks such as Ragas automate scoring later; start by hand.

Contextual retrieval and chunk headers

A chunk cut out of a page loses the context that made it meaningful: which product, which version, which section. Anthropic's contextual retrieval addresses this by asking a model to write a short note, usually 50 to 100 tokens, that situates each chunk in its document, and indexing the note with the chunk. On its own benchmarks, Anthropic reported that this cut failed top-20 retrievals by 35% with embeddings alone, by 49% when combined with BM25, and by 67% with a reranker added.¹

For web pages, start cheaper. The page title and heading path already state most of what a generated note would say. Prepend them to each chunk before embedding (see chunking Markdown for RAG), measure, and only add generated context if your misses show chunks that are still ambiguous with their headers attached. Those numbers are Anthropic's, on its datasets; we have not reproduced them, and your gains will depend on how much context your headers already carry.

Hybrid search: when keywords beat embeddings

Embeddings match meaning; they are weaker on exact strings. Documentation and reference pages are full of exact strings: error codes, function names, version numbers, product names. A user who pastes ERR_CERT_DATE_INVALID wants the page that contains that string, not one about certificates in general.

Hybrid search runs a keyword search (usually BM25) and a vector search, then fuses the two rankings. Two common fusion methods:

• Reciprocal rank fusion (RRF). Each document scores the sum of 1 / (k + rank) over the lists it appears in, with k = 60 in the original paper.² It ignores raw scores, so it needs no calibration.
• Weighted score fusion. Normalise both score lists and mix them with a weight. Weaviate exposes this as an alpha parameter between keyword-only and vector-only.³

Many vector databases support hybrid queries directly. If yours does not, run both searches and fuse with RRF in a few lines of code. Check the effect on your question set; questions with exact terms are where you should see it.

Reranking and top-k

First-stage retrieval compares a query vector with chunk vectors that were computed separately. A cross-encoder reranker reads the query and each candidate together and scores the pair, which is more precise and much slower. The usual pattern, described in the Sentence Transformers retrieve-and-rerank docs, is to retrieve a larger candidate set cheaply and rerank only that.⁴

A reasonable starting configuration: retrieve 50 to 150 candidates with hybrid search, rerank them, and pass the top 10 to 20 to the model. Anthropic's tests used 150 candidates reranked to 20, and found 20 chunks worked better than 5 or 10 in their setup. Your chunk sizes and context budget may point to a different number, so tune k on your question set.

Add a reranker when recall at 50 is high but recall at 5 is low. If recall at 50 is already low, a reranker cannot help; the problem is in chunking or ingestion.

Query rewriting, carefully

Users ask short, vague or multi-part questions. Rewriting can help: expand acronyms, split a two-part question into two queries, or add the product name from the conversation. Keep the user's original wording as one of the queries, because rewrites can drop the exact term that BM25 needed. Generating a hypothetical answer and searching with it (the HyDE approach) is another option; test it against your misses rather than adding it by default.

Packing retrieved context with its sources

How you place chunks in the prompt matters as much as which ones you pick:

• Label each chunk with a short ID, the page title and the URL, so the model can cite it. See RAG citations from web sources.
• Remove near-duplicates before packing. Two copies of the same paragraph waste budget and over-weight one source.
• Group by page and keep document order within a page, so adjacent sections read as one passage.
• Put the strongest evidence at the edges. The "Lost in the Middle" study found that models used information at the beginning and end of a long context more reliably than information in the middle.⁵

Reading failures

When an answer is wrong, check retrieval first: was the right chunk in what the model saw? If not, it is a retrieval or ingestion miss. If it was, it is a generation problem, and the fix is in the prompt, the model or the citation check. Log the retrieved chunk IDs with every answer so this check takes seconds, not an investigation.

Where to go next

If many misses trace back to pages that changed or disappeared after indexing, the fix is in RAG index freshness, not in the retriever.

Footnotes

1. Anthropic, "Introducing Contextual Retrieval" (September 2024). Figures for top-20 retrieval on Anthropic's own datasets. ↩

2. Cormack, Clarke and Büttcher, "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods", SIGIR 2009. ↩

3. Weaviate blog, "Hybrid Search Explained". ↩

4. Sentence Transformers documentation, "Retrieve & Re-Rank". ↩

5. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", arXiv 2307.03172 (2023). ↩