Retrieval
Near-duplicate chunks in the top-k: MMR or dedupe at index time
Stop copies of one paragraph from filling your RAG top-k: tune MMR at query time, remove duplicate chunks at index time, and measure which one your data needs.
By Rose Donaldson, Context Retrieval team · Published
This article was drafted with AI assistance from the sources listed below and edited and published by the Context Retrieval team.
| Criterion | MMR at query time | Dedupe at index time |
|---|---|---|
| Where it runs | At every query | Once, when you build the index |
| Main knobs | fetch_k and lambda_mult | Hash, then MinHash LSH threshold |
| Best for | Paraphrased repeats | Shared text across pages |
A web-sourced index repeats itself more than a curated document set. The same install paragraph sits on a product page, a quick-start and a blog post. A docs site serves one section under two paths. A shared intro opens every page of a guide. When a user asks about that paragraph, the top four results can be four copies of it, and the chunk that held the rest of the answer never reaches the model. This post covers two ways to keep that from happening: maximal marginal relevance (MMR) at query time, and removing duplicate chunks at index time. They are different knobs for the same trade-off, and the evidence that either one helps is thin, so we end with how to test them.
What MMR does to a candidate list
MMR picks results one at a time. For each remaining candidate it computes the similarity to the query, weighted by λ, minus the highest similarity to any chunk already picked, weighted by 1 − λ. The candidate with the best score is added, and the loop repeats.1 A chunk that is very close to something already selected pays for it, even when it is very close to the query too.
LangChain's implementation makes the details concrete. It measures both similarities as cosine similarity between embeddings, picks the single most query-similar candidate first, and then scores the rest with lambda_mult * query_score - (1 - lambda_mult) * redundant_score, where the redundancy score is the maximum similarity to any selected embedding.2 Note what that means for the first result: λ has no effect on it. MMR only changes positions two onward.
Working the score by hand
Take three candidate chunks for the question "How do I install the CLI?" A is the install section of the setup page, B is the same paragraph on a mirror page, and C is the troubleshooting section that covers a failed install. The similarity values below are illustrative, not measured.
| Chunk | Similarity to query | Similarity to A |
|---|---|---|
| A, setup page | 0.90 | 1.00 |
| B, mirror of A | 0.88 | 0.97 |
| C, troubleshooting | 0.80 | 0.40 |
A is picked first. With λ = 0.5, B scores 0.5 × 0.88 − 0.5 × 0.97 = −0.045, and C scores 0.5 × 0.80 − 0.5 × 0.40 = 0.20, so C comes second. With λ = 0.9, B scores 0.792 − 0.097 = 0.695 and C scores 0.72 − 0.04 = 0.68, so the mirror wins by a small margin and the troubleshooting section drops out of a top-2.
def mmr(query_sim, pair_sim, k, lam=0.5):
"""Return candidate indices in MMR order."""
selected = [max(range(len(query_sim)), key=lambda i: query_sim[i])]
while len(selected) < min(k, len(query_sim)):
best, best_score = None, float("-inf")
for i in range(len(query_sim)):
if i in selected:
continue
redundancy = max(pair_sim[i][j] for j in selected)
score = lam * query_sim[i] - (1 - lam) * redundancy
if score > best_score:
best, best_score = i, score
selected.append(best)
return selected
# A: setup page, B: mirror of A, C: troubleshooting section
query_sim = [0.90, 0.88, 0.80]
pair_sim = [
[1.00, 0.97, 0.40],
[0.97, 1.00, 0.42],
[0.40, 0.42, 1.00],
]
print(mmr(query_sim, pair_sim, k=2, lam=0.5)) # [0, 2]
print(mmr(query_sim, pair_sim, k=2, lam=0.9)) # [0, 1]
The lesson from the arithmetic: λ near 1 behaves almost like plain similarity search, and where the pick flips depends on how close your duplicates are to each other and how far the useful alternative trails them. Compute the pairwise similarities of a few known copies in your own index before you pick a λ.
The knobs in LangChain and LlamaIndex
In LangChain, vector stores expose max_marginal_relevance_search. Its defaults are k=4 results, fetch_k=20 candidates passed to the MMR algorithm, and lambda_mult=0.5, a number between 0 and 1 where 0 gives maximum diversity and 1 gives minimum diversity.3 fetch_k matters as much as λ: MMR can only choose among the candidates it was given, so a fetch depth of 20 that holds ten copies of one paragraph leaves little room.
In LlamaIndex, you switch the query mode with vector_store_query_mode="mmr" and pass the weight as vector_store_kwargs={"mmr_threshold": ...}, on a query engine or on a retriever with similarity_top_k.1 The name says threshold, but it plays the role of λ: the docs state that values close to 1 put more emphasis on relevance and values close to 0 on diversity, and that as it approaches 1 the similarity to earlier nodes is ignored.1 The notebook's examples try values from 0.1 to 1.0; it does not give a default, so set it explicitly.
If you also use a reranker, decide the order. A cross-encoder that runs after MMR can put two near-copies back side by side; MMR after the reranker needs the pairwise similarities of the reranked list. The reranker post covers the reranking step itself, and the retrieval guide shows where both sit in the pipeline.
Dedupe at index time instead
MMR treats the symptom at every query. If the repetition comes from the same text on several pages, you can remove it once, when you build the index. Exact copies are cheap to catch: normalize whitespace and case, hash the chunk text, and keep the first chunk per hash.
import hashlib
def chunk_key(text):
normalized = " ".join(text.lower().split())
return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
chunks = [
{"url": "https://example.com/docs/setup", "text": "Install the CLI with npm."},
{"url": "https://example.com/blog/launch", "text": "Install the CLI with npm."},
]
kept = {}
for c in chunks:
key = chunk_key(c["text"])
if key in kept:
kept[key]["also_at"].append(c["url"])
else:
kept[key] = {**c, "also_at": []}
print(list(kept.values()))
The also_at list is the important part. A removed chunk still came from a real page: when that page changes or disappears, your refresh job has to know which stored chunk stood in for it.
Near copies (one changed word, a different footer line) need a similarity test. MinHash LSH finds stored sets whose Jaccard similarity to a query set is above a threshold without comparing against every set; threshold and num_perm are fixed when the index is created. The datasketch docs warn that it returns false positives and can miss qualifying sets, so check each candidate pair before you merge it.4 Chunks from two versions of the same docs page are a special case: they look like duplicates but answer different questions, so keep both and filter by version at query time, as the versioned docs post describes.
Which one, and how to tell
The published evidence does not settle this. The ARAGOG study reported that MMR and Cohere rerank did not show notable advantages over a baseline naive RAG system in its evaluation of retrieval precision and answer similarity.5 That is one study, and it says nothing about an index full of mirrored pages. It is a reason to measure before you turn MMR on by default.
A test that fits in an afternoon:
- Count the problem. Run your golden questions and record, for each, how many of the top-k chunks are exact or near copies of a higher-ranked chunk. If the share is small, stop here.
- Try index-time dedupe if the copies come from shared text across pages. Rebuild, rerun, compare recall at your k.
- Try MMR if the copies are paraphrases rather than shared text. Start with a
fetch_kseveral times your k and λ around 0.5, and read the misses: MMR can push out a second chunk that looked similar but carried the detail the answer needed. - Keep the setting your misses favor, and write down why, so the next person tuning retrieval knows what was tested.
Footnotes
-
LlamaIndex documentation, "Simple Vector Stores: Maximum Marginal Relevance Retrieval". ↩ ↩2 ↩3
-
LangChain source,
langchain_core/vectorstores/utils.py,maximal_marginal_relevance. ↩ -
LangChain Python reference,
max_marginal_relevance_search(Qdrant vector store). ↩ -
datasketch documentation, "MinHash LSH". ↩
-
Eibich, Nagpal and Fred-Ojala, "ARAGOG: Advanced RAG Output Grading", arXiv 2404.01037 (April 2024). ↩
Frequently asked questions
What is maximal marginal relevance in RAG?
MMR is a way of picking retrieved chunks one at a time. Each pick scores a candidate by its similarity to the query minus its highest similarity to the chunks already picked, weighted by a parameter lambda, so a chunk that repeats an earlier pick loses points.
What do fetch_k and lambda_mult mean in LangChain?
fetch_k is the number of documents fetched and passed to the MMR algorithm, 20 by default. lambda_mult is a number between 0 and 1, 0.5 by default, where 0 gives maximum diversity and 1 gives minimum diversity. k, the number of documents returned, defaults to 4.
Does MMR improve RAG answers?
Not reliably. The ARAGOG study reported that MMR did not show notable advantages over a baseline naive RAG system on its evaluation. Whether it helps depends on how many near-duplicate chunks your index has, so measure it on your own question set.
Should I remove duplicate chunks before embedding instead?
If the repetition comes from the same text appearing on several pages, removing exact and near-duplicate chunks at index time fixes the cause and stores fewer vectors. Keep a record of every URL the removed copies came from, so you can still cite or delete them later.
Sources
- max_marginal_relevance_search (LangChain Python reference, Qdrant) (Documentation)
- langchain_core/vectorstores/utils.py: maximal_marginal_relevance (LangChain source) (Documentation)
- Simple Vector Stores: Maximum Marginal Relevance Retrieval (LlamaIndex docs) (Documentation)
- ARAGOG: Advanced RAG Output Grading (arXiv 2404.01037) (Paper)
- MinHash LSH (datasketch docs) (Documentation)