BEIR: Zero-Shot Retrieval Evaluation for RAG

BEIR is the benchmark that measures the retrieval half of a RAG pipeline: how well a model finds relevant documents in a domain it was not trained on. Eighteen datasets, nine task types, nDCG@10 as the headline number. It is the first check for retrieval generalisation, and it deliberately says nothing about the generated answer.

BEIR, introduced by Thakur et al. in 2021 and accepted to the NeurIPS 2021 Datasets and Benchmarks track, is a heterogeneous information-retrieval benchmark built to test one thing that earlier benchmarks did not: how well a retriever generalises to domains and query styles it was never trained on. Each task gives the model a query and a document corpus; the model ranks the corpus, and the ranking is scored against human relevance judgements. There is no generation step and no answer to grade, only a ranked list.

In a retrieval-augmented-generation pipeline, that ranked list is exactly the retrieval stage: the documents that get passed to the model as context. BEIR therefore measures the input quality of RAG. A high BEIR score means the retriever surfaces the right context; it says nothing about whether the downstream model then uses that context faithfully. This is why honest RAG claims never quote BEIR alone: retrieval quality and generation faithfulness are separate axes, and BEIR covers only the first.

The finding that made BEIR influential was uncomfortable for the dense-retrieval literature of the time. Neural retrievers that comfortably beat the BM25 lexical baseline in-domain frequently lost to it out-of-domain. BEIR quantified that generalisation gap across eighteen datasets, and in doing so it changed how retrieval models are reported: a model card that quotes only in-domain MS MARCO numbers is now treated as incomplete.

BEIR's eighteen datasets are grouped into nine task types spanning open-domain QA, scientific and biomedical retrieval, fact-checking, argumentation, and more. The breadth is the point: a model that scores well across all nine has demonstrated retrieval competence that transfers, not competence tuned to one corpus.

Not all eighteen datasets are freely downloadable. TREC-NEWS, Robust04, Signal-1M and BioASQ are licence-gated or require manual retrieval, so most public evaluations report an average over the fifteen to seventeen datasets they can access. When you compare two BEIR figures, confirm they cover the same subset; a mean over thirteen easy datasets is not the same benchmark as a mean over all eighteen.

The BEIR framework computes a full spread of retrieval metrics (nDCG, MAP, Recall and Precision at k values from 1 to 1000, plus MRR), but the community settled on the mean nDCG@10 across datasets as the headline. nDCG@10 rewards placing relevant documents in the top ten results and discounts relevance further down the list, which mirrors how a RAG pipeline actually uses retrieval: only the first handful of retrieved passages usually make it into the context window.

Recall@k matters when your pipeline retrieves a large candidate set and re-ranks it later; in that design a high Recall@100 with a modest nDCG@10 can be fine, because the re-ranker fixes the ordering. Choose the metric that matches your architecture: nDCG@10 for retrieve-then-read pipelines, Recall@k for retrieve-then-rerank pipelines. Quoting nDCG@10 by default is reasonable, but say so, because a bare "BEIR score" is ambiguous about which k and which metric.

BEIR and MTEB are often quoted together and are easy to confuse. BEIR is retrieval-only: eighteen datasets, one job, nDCG@10. MTEB (the Massive Text Embedding Benchmark) is broader, covering many datasets across eight task types including classification, clustering, reranking, semantic similarity and retrieval. MTEB's retrieval track incorporates BEIR datasets directly, so the two overlap on retrieval and diverge everywhere else.

The practical rule: use BEIR when the question is narrow (how good is this retriever at finding relevant documents across domains), and use MTEB when the question is broad (how good is this embedding model for all the jobs a production pipeline asks of it, from retrieval to reranking to deduplication). Most current embedding model cards report both; when comparing models, quote both and read the retrieval sub-scores rather than only the top-line average.

Three caveats travel with every BEIR number. First, saturation: several individual datasets are close to solved for frontier embedding models, so the headline mean compresses and one-point gaps stop being meaningful. Read the per-dataset table, and weight the datasets that match your domain. Second, subset drift: because of the licence-gated datasets, published BEIR averages silently cover different dataset sets, which makes cross-paper comparison unreliable unless the subset is stated. Third, the zero-shot label is model-relative: MS MARCO and other in-domain datasets are not zero-shot for a model trained on them, so their contribution to that model's average overstates its generalisation.

None of these break BEIR as a research artefact; they are reasons to read it carefully rather than quote the mean. The wider issues of benchmark contamination and what benchmarks miss apply to retrieval evaluation as much as to reasoning benchmarks: a retriever that has seen a corpus in pre-training is not being tested zero-shot on it.

Reach for BEIR first when you are selecting a retriever or embedding model and you care about generalisation to new domains, which in RAG you almost always do. Quote the mean nDCG@10 plus the per-dataset breakdown, name the dataset subset, and pair it with MTEB for the broader embedding picture. Then, because BEIR stops at retrieval, add an end-to-end benchmark (RAGAS or a golden dataset scored with LLM-as-judge) to check that the generation stage uses the retrieved context faithfully. See our RAG evaluation deep dive for how the retrieval and generation metrics fit together.