Rerankers for RAG: a cross-encoder first, an LLM only where it pays

A reranker is the second pass of retrieval: the first stage returns a few dozen candidate chunks cheaply, and the reranker reads them again more carefully before the best ones reach the model. The retrieval guide says when to add one: when the right chunk is usually in your top 50 but not in your top 5. This post goes one step further and asks which kind of reranker to add. The short answer is a cross-encoder first. An LLM reranker is a later experiment, run on a slice of traffic, with its latency written down before you start.

Two encoders, two jobs

A bi-encoder embeds queries and passages independently. That is what makes vector search fast: passage embeddings are computed once, at index time. A cross-encoder takes the query and one passage together and returns a single relevance score. Because attention runs across both texts, it is more accurate, and it is also too slow to score every pair in a large collection.¹

The Sentence Transformers docs describe the resulting pattern: retrieve a candidate set with a bi-encoder or lexical search, then rerank only those candidates with a cross-encoder. Their example retrieves about 100 candidates.¹ For a web-sourced index, the first stage is often the hybrid BM25 and vector search described in the reciprocal rank fusion post, and the reranker reorders the fused list.

A cross-encoder in a few lines

The CrossEncoder class has a rank(query, documents) method. It returns a list sorted by score, each entry holding a corpus_id (the index of the document in your input list) and a score. top_k defaults to None, which returns every document, and batch_size defaults to 32.² The usage docs load cross-encoder/ms-marco-MiniLM-L6-v2 as their example model and note that MS MARCO models return logits, not scores between 0 and 1. A sigmoid activation maps them into that range without changing the order.³

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

query = "How do I rotate an API key?"
candidates = [
    {"url": "https://example.com/docs/keys", "title": "API keys",
     "path": "Security > Rotating keys",
     "text": "Create a new key, deploy it, then revoke the old one."},
    {"url": "https://example.com/docs/billing", "title": "Billing",
     "path": "Invoices > Download",
     "text": "Invoices are available as PDF in the dashboard."},
    {"url": "https://example.com/blog/release-2", "title": "Release notes",
     "path": "Changes > Keys",
     "text": "Keys now show the date they were last used."},
]

# Score the chunk with the context a reader would have: title and heading path.
docs = [f"{c['title']} | {c['path']}\n{c['text']}" for c in candidates]

for hit in model.rank(query, docs, top_k=2):
    c = candidates[hit["corpus_id"]]
    print(f"{hit['score']:.2f}  {c['url']}  ({c['path']})")

Because rank returns positions, not text, the URL, title and heading path stay attached to each chunk through the reranking step. The prompt needs them later for citations.

What a web chunk should show the reranker

A reranker scores only the text you give it. A chunk cut from the middle of a documentation page often lacks its subject: "Create a new key, then revoke the old one" does not say which product or which section it belongs to. The example above prepends the page title and heading path, which you already have if you chunked the fetched Markdown by headings. That is a design choice, not a measured result. Test it on your question set against the bare chunk text.

Two limits are worth checking. The CrossEncoder constructor takes a max_length,² and a long chunk with a long prefix can be cut before the part that answers the question. And the candidate count sets the cost: every candidate is one more forward pass, in batches of batch_size.

What an LLM reranker cost one team

Fin's research team published a detailed account in September 2025. The authors found that open-source cross-encoders, with the BGE reranker as their baseline, did not reach their quality bar, so they built a pointwise LLM reranker that scores each passage from 1 to 10.⁴ They scored 40 candidates per query, split into 4 batches of 10 that run in parallel.

Their first working version added about 5 seconds of latency. A more compact output format (about 28% fewer output tokens), dropping passages scored below 5, prompt caching and parallel calls brought the added latency under 1 second, and they report costs fell about 8x.⁴ In an A/B test against BGE, the LLM reranker gave a statistically significant lift in resolution rate. They tried listwise reranking too: the same resolution rate, with about 40% more latency and about 15% more cost. A shard that times out falls back to BGE scores.⁴

Even optimized, the added P50 latency was +0.9 seconds.⁴ So the team used the LLM as a teacher. Its follow-up post describes a cross-encoder built on ModernBERT-large and trained on LLM labels for 400,000 real queries with 40 candidates each, 16 million pairs.⁵ Fin gives a P50 latency of about 150 ms, unchanged from the Cohere baseline in its A/B test, and reports 80% lower reranking costs.⁵

These are one company's numbers, on its own data and traffic, and they are not reproduced here. What transfers is the shape of the path: the LLM reranker won on quality and lost on latency, and the team ended up back at a cross-encoder.

A decision rule you can apply

1. Measure recall at the candidate depth. If the right chunk is missing from the top 50, no reranker helps. Look at chunking and ingestion instead.
2. Add a cross-encoder and compare recall at the number of chunks you pass to the model, before and after.
3. Write down your latency budget. If an extra second per answer is unacceptable, an LLM reranker in the request path is out, whatever it scores.
4. If misses remain and the budget allows, run an LLM reranker on a slice of questions, with thresholding and a cross-encoder fallback on timeout, as Fin did.
5. If the LLM clearly wins, its scores are training labels. Fine-tuning a cross-encoder on them is the route Fin published. Whether it pays at your query volume is a question for your own numbers.

Footnotes

1. Sentence Transformers documentation, "Retrieve & Re-Rank". ↩ ↩²

2. Sentence Transformers documentation, CrossEncoder API reference (rank, predict and constructor parameters). ↩ ↩²

3. Sentence Transformers documentation, "Cross-Encoder usage". ↩

4. Ramil Yarullin and Fedor Parfenov, "Using LLMs as a Reranker for RAG: A Practical Guide", Fin research, 11 September 2025. ↩ ↩² ↩³ ↩⁴

5. "How We Built a World-Class Reranker for Fin", Fin research, September 2025. ↩ ↩²