Re-ranking

Re-ranking is a two-stage retrieval pattern: a fast retriever proposes candidates, then a slower, stronger model scores and reorders them by relevance.

First-stage retrieval with BM25, dense retrieval, or a hybrid can scan millions or billions of documents and return the top 100-1000. A re-ranker, usually a cross-encoder transformer, then scores each candidate by attending to the query and document together. Cross-encoders beat bi-encoders on fine interactions, but they are too slow to run over a full corpus.

Restricting them to a shortlist keeps coverage and raises precision.

The re-ranker outputs a relevance score per pair, using detail, context, and semantic match. Cohere's Rerank, BGE Reranker, and other cross-encoders train on relevance judgments. In RAG, the chunks that reach the LLM decide answer quality, so this second pass is worth the extra compute. Retrieval then re-rank is the usual production search stack.

Stage one is recall: BM25, dense, or hybrid over the whole corpus, top 100-1000. Stage two is precision: a cross-encoder concatenates query and document and outputs one relevance score. Bi-encoders cannot match that token-level interaction, which is why they are not used alone for the final order. Cohere Rerank and BGE Reranker are trained on labeled relevance pairs.

RAG uses the new order to pick chunks for the LLM. Production search keeps this two-step shape because the expensive model never scans billions of docs. A cheap retriever fetches 100 hits. A slower model re-ranks them. That two-stage setup is standard in search and RAG.