HotpotQA: Multi-Hop Question Answering With Checkable Evidence

The foundational multi-hop QA benchmark. 113,000 Wikipedia questions that cannot be answered from a single document, each shipped with the exact sentences the answer depends on so the reasoning chain is checkable, not just the final answer. The benchmark that made supporting-fact supervision standard.

HotpotQA, introduced by Yang et al. at EMNLP 2018, tests whether a system can answer questions that require combining information from more than one source. Its roughly 113,000 crowd-sourced questions are each built from a pair of Wikipedia articles, and by construction the answer cannot be found in either article alone: the system has to chain a fact from the first to reach the answer in the second. That two-hop requirement is what separates HotpotQA from single-document reading-comprehension datasets like SQuAD, which a model can often solve by pattern-matching a span in one passage.

The benchmark's defining contribution is sentence-level supporting facts. Every question is annotated with the specific sentences needed to justify the answer, and a system is scored on whether it identifies those sentences as well as on whether it produces the right answer. The authors call this "explainable" multi-hop QA: the evidence is a first-class output, not an afterthought. This design discourages the shortcut where a model guesses a plausible answer without doing the reasoning, because a correct answer with the wrong evidence loses joint credit.

HotpotQA also introduced a category of factoid comparison questions, which ask a system to extract a property from two entities and compare them. These sit alongside the more common bridge questions and stress a different skill: not following a chain of retrieval, but pulling parallel facts and reasoning over the pair. Together the two question types make HotpotQA a broader test of multi-hop reasoning than a bridge-only dataset would be.

HotpotQA is evaluated in two settings that separate reasoning from retrieval. They report as two different leaderboards and are not comparable to each other, so any quoted HotpotQA number should say which setting it came from.

The distractor setting is the closer analogue to a reading-comprehension task: the answer is somewhere in the 10 supplied paragraphs, and the 8 distractors are there to punish sloppy attention. The fullwiki setting is the RAG-shaped one, and it is where multi-hop is genuinely hard: the system must retrieve the right two documents out of millions before it can reason at all, so a single retrieval miss on the first hop makes the second hop impossible. This is exactly the failure mode a production retrieval pipeline has to survive, which is why fullwiki is the more informative setting for RAG work.

The two question types exercise different parts of the multi-hop skill and are worth reading separately.

Comparison questions are the ones most exposed to lucky guessing, since many reduce to a yes/no or a choice between two named entities. This is the clearest argument for the supporting-fact score: a model that answers "yes" correctly but cites the wrong sentences has not demonstrated the reasoning, and the joint metric refuses it full credit. Reading answer-only accuracy on the comparison subset without the supporting-fact score overstates how much genuine multi-hop reasoning is happening.

HotpotQA scores two things and combines them. The answer score is exact-match (EM) and token-level F1 against the gold answer. The supporting-fact score is EM and F1 against the annotated supporting sentences. The joint EM and joint F1 combine the two so that a system only earns full joint credit when it gets both the answer and its evidence right. The joint number is the one that reflects the benchmark's intent; the answer-only number is easier and less honest to quote alone.

Because the supporting facts are annotated at the sentence level, the benchmark rewards models that ground their answers in specific evidence rather than in a diffuse read of the whole context. That property has aged well: it is the same idea that citation-grounded RAG systems in 2026 are built around, which is part of why HotpotQA is still taught even though its raw answer scores have climbed. See our RAG evaluation reference for how faithfulness and grounding are measured in the generation stage more broadly.

HotpotQA is foundational, not frontier. Top distractor-setting systems now reach the low-to-mid 70s in answer EM and the fullwiki board sits in the high 60s (official leaderboard, checked August 2026), so the benchmark no longer separates the strongest current systems the way it did at launch. It is also Wikipedia-only and predominantly two-hop, which caps how far it can stress the long retrieval chains that harder modern questions demand. We do not reprint a per-model leaderboard table here: HotpotQA maintains its own live boards for the distractor and fullwiki settings, and those are the honest place to read current standings with the setting attached.

For evaluating a production retrieval-augmented pipeline on multi-hop reasoning, MultiHopRAG (Tang et al., 2024) is the more directly applicable benchmark, because it is designed around the RAG workflow and evaluates which retrieved documents the answer depends on. HotpotQA remains the reference everyone cites, the cleanest teaching example of supporting-fact supervision, and a stable trend line against older systems. Read the two together: HotpotQA for the concept and the history, MultiHopRAG for the current RAG-shaped measurement. See how both fit the wider picture on our RAG benchmarks comparison.

Reach for HotpotQA when you want a well-understood, widely-cited multi-hop QA benchmark with checkable evidence: for teaching or explaining multi-hop reasoning, for a trend line against 2019-2024 systems, or for stress-testing whether a system grounds its answers in the right sentences rather than guessing. Use the fullwiki setting, not just distractor, if retrieval is part of what you are evaluating, and always quote the joint EM and F1 alongside the answer-only score. For a production RAG pipeline on multi-hop queries prefer MultiHopRAG; for pure retrieval quality use BEIR; for embedding quality use MTEB; for end-to-end RAG faithfulness use RAGAS.