Document Retrieval

Document retrieval finds relevant documents for a query. It is the first stage of search engines and retrieval-augmented generation.

TF-IDF and BM25 score lexical overlap: documents that contain query terms rank higher, with weights for term importance and document length. They are fast and readable, but they miss paraphrases. Dense retrieval embeds queries and documents in one vector space and returns nearest neighbors.

A query about "machine learning" can retrieve documents about "ML" and "neural networks" with no shared string. The embedding model trains on query-document relevance so relevant documents sit close to their queries. Hybrid retrieval mixes lexical and dense scores and often beats either alone.

The pipeline indexes document embeddings, generates candidates with approximate nearest neighbor search, then optionally re-ranks. NDCG, MRR, and Recall@k measure whether the right documents sit near the top. RAG fails if retrieval misses the right documents, no matter how good the generator is.

BM25 is the lexical baseline: term frequency, inverse document frequency, length penalty. " Hybrid adds both scores. Indexing precomputes document vectors. Approximate nearest neighbor search returns a candidate list fast. Re-ranking can follow. NDCG, MRR, and Recall@k ask whether the relevant docs are near rank 1, not merely somewhere in the corpus.

RAG generators cannot cite a document the retriever never returned. Classic IR ranks documents for a query. Vector search is the same job with embeddings instead of only keyword indexes.