MS MARCO: Ranking Passages Against Real Bing Queries

The canonical large-scale passage-ranking benchmark. 8.8 million passages and roughly half a million real Bing search queries, scored with MRR@10. Nearly every dense retriever in 2026 is trained on it, which is both its strength and the reason an out-of-domain check on BEIR is the standard companion.

MS MARCO, introduced by Nguyen et al. at Microsoft in 2016, was the first retrieval dataset built at genuine web scale from real user behaviour. The team sampled 1,010,916 anonymised questions from Bing's search query logs, collected the web documents Bing returned for them, and extracted a corpus of 8,841,823 passages from 3,563,535 documents. Because the queries are real searches rather than constructed ones, the distribution reflects how people actually ask questions, which is part of why the dataset became the field's default training set for retrieval.

The task most people mean by "MS MARCO" is passage ranking: given a query, order the 8.8-million-passage collection so that relevant passages come first. It is a pure retrieval task with a programmatic success function, which makes it reproducible and cheap to evaluate at scale, the opposite trade-off from an LLM-as-judge end-to-end RAG score. That reproducibility, combined with roughly 500,000 labelled training queries, is what let the dense-retrieval era train on it: bi-encoders, cross-encoders, and the retrieval components inside modern RAG stacks are overwhelmingly tuned on MS MARCO.

MS MARCO supports three related tasks off the same underlying data. They are frequently conflated, so any quoted number should say which task it came from.

Passage ranking is the headline: it has the public leaderboard, the largest body of published results, and the cleanest programmatic scoring. Document ranking is the coarser-grained sibling, ranking whole documents rather than extracted passages, and shows up mostly through the TREC Deep Learning track. Question answering is the original 2016 framing: read the retrieved passages and produce (or judge the answerability of) a human-style answer. When a paper says "MS MARCO" with no qualifier, it almost always means passage ranking.

The official passage-ranking metric is MRR@10, the mean reciprocal rank of the first relevant passage within the top 10 results. If the first relevant passage sits at rank 1 the query scores 1.0; at rank 2 it scores 0.5; beyond rank 10 it scores 0. MRR@10 was chosen because the public dev and eval sets have sparse relevance judgements, often a single known-relevant passage per query, and a rank-of-first-hit metric behaves sensibly under that sparsity where a recall-oriented metric would not.

The sparse-judgement design is also MS MARCO's main measurement caveat. With roughly one labelled passage per query, a system can retrieve a genuinely relevant passage that simply was not annotated and be scored as wrong. The TREC Deep Learning track exists partly to address this: it reuses the MS MARCO corpora but collects dense human relevance judgements on a smaller query set and reports NDCG@10, which is the more informative metric when full judgements are available. Read MRR@10 off the MS MARCO leaderboard; read NDCG@10 off TREC Deep Learning.

There are two generations of the corpus, and quoting the wrong one confuses scale comparisons.

v2, introduced for the TREC 2021 Deep Learning track, is document-native: it began from 11.9 million documents (2.7 million surviving v1 URLs plus 9.2 million new ones) and segmented them into 138 million passages, an average of 11.6 passages per document, with an explicit passage-to-document mapping so participants can use document context in passage ranking and vice versa. Despite the larger, cleaner corpus, most published dense-retrieval results still cite v1 passage ranking, because that is where the training data, the baselines, and the leaderboard history live. v2 is the right corpus for TREC Deep Learning work and for stress-testing retrieval at larger scale.

The single most important thing to understand about MS MARCO in 2026 is its relationship to BEIR. MS MARCO is the in-domain workhorse: because it is the large labelled set most retrievers train on, a strong MS MARCO score partly measures how well a model fits the exact distribution it was trained on. That is useful, but it says little about whether the model generalises to biomedical search, legal retrieval, or fact-checking, domains a production system will actually meet.

BEIR was built as the answer to that gap: an 18-dataset zero-shot suite that measures out-of-domain retrieval, with MS MARCO included as one of its datasets. The overfitting failure mode is real and documented, a retriever tuned aggressively on MS MARCO can top the in-domain leaderboard and then lose to a simpler model on BEIR's out-of-domain tasks. The honest 2026 pattern is to report both: MS MARCO for in-domain passage ranking, BEIR NDCG@10 for generalisation. Quoting only the in-domain number is the retrieval equivalent of reporting training accuracy.

Reach for MS MARCO when you are training or evaluating a retriever and want the standard in-domain benchmark with a decade of comparable baselines: fine-tuning a dense retriever, comparing rerankers, or reporting a passage-ranking number readers can place against published results. Quote MRR@10 with the version attached, prefer v1 for comparability, and always pair it with an out-of-domain BEIR number so the score is not read as generalisation. For embedding-model selection across many task types use MTEB; for multi-hop reasoning use HotpotQA or MultiHopRAG; for end-to-end RAG faithfulness use RAGAS. MS MARCO measures one stage, retrieval, very well, and the honest claim names that stage.