MedQA: 12,723 USMLE Questions, Med-PaLM 2 at 86.5% Accuracy

The benchmark that put medical LLM reasoning on the map. The 2026 frontier has moved on.

The MedQA dataset draws multiple-choice questions from the USMLE step exam practice pools and equivalent Chinese and Taiwanese licensing exams. Each question presents a clinical vignette (patient history, exam findings, lab results) and asks for the most likely diagnosis, the next investigation, or the appropriate treatment. The questions are standardised, well-grounded in clinical practice, and authored by medical educators.

Accuracy is the headline metric, computed against the gold answer. There is no partial credit and no judge. The English (USMLE) subset has 12,723 questions; reported scores almost always use this subset unless explicitly noted.

Three published results map the benchmark's arc. The original Jin et al. paper (2020) reported a best pretrained baseline (BioBERT-Large) of 36.7% on MedQA-USMLE. The Google Flan-PaLM 540B work (Dec 2022) reached 67.6%, the first broadly general model to enter passing-USMLE territory. Med-PaLM 2 (Singhal et al., 2023) reported 86.5%, clearing the expert-passing threshold often cited for the exam. All three figures come from the primary papers listed in Sources below.

Beyond those published milestones we do not reprint a current per-model MedQA score table. The benchmark is near saturation for frontier models, there is no maintained official leaderboard tracking it, and vendor-reported medical numbers come from harnesses we cannot inspect, so any current table we copied would be unverifiable. For frontier discrimination, harder successors like MedXpertQA are now used; start from the homepage task picker to find the benchmark that matches your use case.

MedQA is most often quoted on its own, but it is also one component of MultiMedQA, the suite Google assembled to evaluate the Med-PaLM lineage. MultiMedQA combines seven question-answering datasets so that a single model is tested on licensing-exam reasoning, research comprehension, and free-text consumer health answers together, rather than on any one of them in isolation. The free-text components are scored by clinicians, which is what let the Med-PaLM papers report qualitative safety judgements alongside the multiple-choice accuracy numbers.

Because MedQA-USMLE now sits near saturation, the useful frontier signal in medicine has shifted to harder successors. MedXpertQA (arXiv:2501.18362, ICML 2025) was built specifically to restore discrimination between top models: it is smaller but much harder, spans more specialties, and adds a multimodal subset so that image-grounded clinical reasoning is tested rather than text alone. If you are choosing a medical benchmark in 2026, MedQA is the historical anchor and MedXpertQA is the discriminating current test.

MedQA accuracy is a measure of multiple-choice exam performance. It is not a measure of clinical safety, hallucination rate, citation accuracy, or fitness for medical advice. The Google Med-PaLM 2 paper is explicit on this point: the model required clinician review for free-text consumer answers, and the 86.5% number does not imply 86.5% safe medical advice. Any deployment of an LLM for medical purposes is bound by FDA, MHRA, or EMA regulation depending on jurisdiction. A benchmark score is not a regulatory clearance.