Retrieval
BM25 vs embeddings on documentation pages: which queries each wins
BM25 finds exact strings such as error codes and versions; embeddings find paraphrases. On docs pages you usually need both, fused into one ranking.
Published
This article was drafted with AI assistance from the sources listed below and edited and published by Teneo Labs.
| Criterion | BM25 | Embeddings |
|---|---|---|
| Error code or exit status | Strong: exact token match | Weak: codes carry little meaning |
| Function or setting name | Strong | Mixed: depends on the model |
| Version number | Strong | Weak |
| Paraphrased how-to question | Weak: different words | Strong |
| Concept with synonyms | Weak | Strong |
Keyword search and vector search fail on different questions. On documentation pages, which are full of identifiers, that difference decides whether a user who pastes an error message gets the right page.
What each one matches
BM25 scores a chunk by how often the query's terms appear in it, weighted by how rare each term is across the index. A rare string such as ERR_CERT_DATE_INVALID or max_retries gets a high weight, so the chunks that contain it rise to the top.
An embedding model maps the query and each chunk to vectors and compares them. It matches meaning, so "how do I stop retrying failed calls" can find a section titled "Retry policy" that shares no words with the question. The same property makes it weaker on identifiers: an error code carries little meaning for the model to match.
What the benchmarks say
The BEIR benchmark compared retrieval models zero-shot across 18 datasets from different domains. BM25 proved a strong baseline, and several dense retrievers trained on one dataset fell below it on others.1 Embedding models have improved since, but the lesson holds for specialised corpora: a model has not necessarily seen your product's vocabulary.
Anthropic's contextual retrieval write-up reports that combining embeddings with BM25 reduced failed retrievals compared with embeddings alone on its own datasets.2 Those are its numbers on its data; we have not reproduced them.
Use both, then check
Hybrid search runs both and fuses the rankings, either with reciprocal rank fusion or with a weighted score that many databases expose as a single parameter, such as Weaviate's alpha.3 The retrieval guide covers both fusion methods.
To check whether you need it, tag each question in your evaluation set by type: identifier, paraphrase or concept. Measure recall at k per type with keyword-only, vector-only and hybrid retrieval. If identifier questions miss with vectors alone, hybrid search is the fix; if paraphrases miss with keywords alone, keep the embeddings.
Two details that change the result
- Tokenisation. Many BM25 analysers split on punctuation, so
max_retriesmay become two tokens. Index identifiers in a field that keeps them whole, or test how your analyser handles them. - Headers in the text. If you prepend the page title and heading path to each chunk, as the chunking guide suggests, BM25 can match product and section names too.
Footnotes
Frequently asked questions
Do I still need BM25 if I use a good embedding model?
On documentation and reference pages, usually yes. Questions with exact identifiers such as error codes, setting names and version numbers are where keyword search helps most. Check your own question set: if those questions miss, add BM25.
Which is cheaper to run, BM25 or vector search?
BM25 needs no model calls at indexing or query time. Vector search needs an embedding per chunk and per query. Running both adds the cost of the keyword index, which is small next to the embeddings.