Abstract

What12,723 USMLE-style multiple-choice medical questions (English subset most cited).

WhoJin, Pan, Oh, Lu, Welleck, Cohen, Lu, Szolovits (2020).

Published milestoneMed-PaLM 2 at 86.5% on MedQA-USMLE (Singhal et al., 2023); benchmark now near saturation.

Paperarxiv.org/abs/2009.13081

Section I.viii Industry Domain|Reviewed 2026

MedQA: 12,723 USMLE Questions, Med-PaLM 2 at 86.5% Accuracy

The benchmark that put medical LLM reasoning on the map. The 2026 frontier has moved on.

Construction

The MedQA dataset draws multiple-choice questions from the USMLE step exam practice pools and equivalent Chinese and Taiwanese licensing exams. Each question presents a clinical vignette (patient history, exam findings, lab results) and asks for the most likely diagnosis, the next investigation, or the appropriate treatment. The questions are standardised, well-grounded in clinical practice, and authored by medical educators.

Accuracy is the headline metric, computed against the gold answer. There is no partial credit and no judge. The English (USMLE) subset has 12,723 questions; reported scores almost always use this subset unless explicitly noted.

Published Milestones

Three published results map the benchmark's arc. The original Jin et al. paper (2020) reported a BioBERT baseline at 38.1% on MedQA-USMLE. The Google Flan-PaLM 540B work (Dec 2022) reached 67.6%, the first broadly general model to enter passing-USMLE territory. Med-PaLM 2 (Singhal et al., 2023) reported 86.5%, clearing the expert-passing threshold often cited for the exam. All three figures come from the primary papers listed in Sources below.

Beyond those published milestones we do not reprint a current per-model MedQA score table. The benchmark is near saturation for frontier models, there is no maintained official leaderboard tracking it, and vendor-reported medical numbers come from harnesses we cannot inspect, so any current table we copied would be unverifiable. For frontier discrimination, harder successors like MedXpertQA are now used; start from the homepage task picker to find the benchmark that matches your use case.

III

YMYL Reading Note

MedQA accuracy is a measure of multiple-choice exam performance. It is not a measure of clinical safety, hallucination rate, citation accuracy, or fitness for medical advice. The Google Med-PaLM 2 paper is explicit on this point: the model required clinician review for free-text consumer answers, and the 86.5% number does not imply 86.5% safe medical advice. Any deployment of an LLM for medical purposes is bound by FDA, MHRA, or EMA regulation depending on jurisdiction. A benchmark score is not a regulatory clearance.

LegalBench for legal reasoning →RepoBench for code reasoning →Harm and safety evals →

Reader Questions

Q.01What is MedQA?+

MedQA is a benchmark of 12,723 free-form multiple-choice medical questions sourced from the US Medical Licensing Examination (USMLE) Step 1, Step 2 CK, and Step 3 practice materials, plus Chinese and Taiwanese equivalents. The English subset (USMLE-only, often called MedQA-USMLE) is the most cited. The benchmark tests factual recall, diagnostic reasoning, and treatment selection.

Q.02What is MultiMedQA?+

MultiMedQA is a Google-curated suite of 7 medical question-answering datasets combined for evaluating the Med-PaLM lineage. It includes MedQA, MedMCQA, PubMedQA, MMLU clinical-topics subsets, LiveQA, MedicationQA, and HealthSearchQA. The benchmark introduced 'consumer health' free-text answers alongside the multiple-choice tests, evaluated by clinicians on 12 dimensions.

Q.03What was the Med-PaLM 2 headline?+

The Singhal et al. paper (Google, 2023) reported Med-PaLM 2 at 86.5% accuracy on MedQA-USMLE, the first model to clear the 85% expert threshold often cited as a passing USMLE benchmark. The same model scored well on MedMCQA and PubMedQA but the team reported it required clinician oversight for free-text consumer answers.

Q.04Are 2026 frontier models reliable for medical work?+

Benchmark scores have risen, but real-world deployment in medicine remains subject to regulatory oversight (FDA in the US, MHRA in the UK, EMA in the EU). A high MedQA score is not a clinical safety claim. Models can still hallucinate drug interactions, miss rare presentations, and fail under prompt-template variance. Reading a model card's MedQA number as 'safe to use' is exactly the misreading that the YMYL ranking guidelines exist to prevent.

Q.05Is MedQA at risk of saturation?+

MedQA-USMLE is widely regarded as approaching saturation: strong general models now clear the expert-passing threshold without medical fine-tuning, so the benchmark no longer discriminates well between frontier models. The harder MedXpertQA (released 2024) was built specifically to restore that discrimination and is the more useful frontier benchmark in medicine today. PubMedQA is also approaching saturation and is increasingly used as a literature-comprehension probe rather than a frontier signal. For current per-model numbers, consult each benchmark's own reporting rather than a figure reprinted here.

Sources

[1] Jin et al. (2020): arxiv.org/abs/2009.13081
[2] Singhal et al., Med-PaLM 2 (2023): arxiv.org/abs/2305.09617
[3] MultiMedQA introduction (Singhal et al. 2022): arxiv.org/abs/2212.13138
[4] MedXpertQA (2024): arxiv.org/abs/2501.18362