# AI Benchmarks 2026: Which Benchmark to Use and How to Read It > Frontier models, agent evals (SWE-bench, WebArena), RAG metrics, custom pipelines. Which benchmark fits which task, where each breaks, and how to read the scores. Reprinted rows are dated to their... ## Pages - [Home](https://benchmarkingagents.com/): Frontier models, agent evals (SWE-bench, WebArena), RAG metrics, custom pipelines. Which benchmark fits which task, where each breaks, and how to read... - [About benchmarkingagents.com: Independent AI Benchmark Reference](https://benchmarkingagents.com/about/): Who builds benchmarkingagents.com, why it exists, what it covers, editorial principles, and how to contact for corrections. Independent reference for AI agent... - [AI Agent Benchmarks - SWE-bench, WebArena, AgentBench, Terminal-Bench, OSWorld, Tau-Bench](https://benchmarkingagents.com/agent-benchmarks/): Comprehensive guide to AI agent benchmarks. SWE-bench Verified, WebArena, AgentBench, Terminal-Bench, OSWorld, and Tau-Bench explained with methodology, construction, and honest limitations. Per-model... - [Agent Eval Cost Calculator: Estimate $ to Run SWE-bench, GAIA, TauBench](https://benchmarkingagents.com/agent-eval-cost-calculator/): Order-of-magnitude estimates of the dollar cost to run popular agent benchmarks end-to-end, worked from public token counts and current Anthropic list pricing.... - [AgentBench: Methodology, Eight Environments, Coverage Map](https://benchmarkingagents.com/agentbench/): AgentBench evaluates LLMs as autonomous agents across eight environments. Methodology, what the eight environments cover, why the headline mean misleads, and where... - [AndroidWorld Benchmark: 116 Mobile UI Tasks Across 20 Android Apps](https://benchmarkingagents.com/androidworld/): AndroidWorld leaderboard as read 18 Nov 2026: five self-reported entries at 100% pass@1; the sheet was archived as saturated in Nov 2026. 116 tasks across 20 Android apps; the 2024 paper baseline was 30.6%. - [AppWorld Benchmark: 750 Tasks Across 9 Apps, Code-as-Action](https://benchmarkingagents.com/appworld-benchmark/): AppWorld (Trivedi et al., ACL 2024) tests autonomous agents on 750 multi-app tasks against 9 simulated services. Methodology, the TGC vs SGC... - [AssistantBench: 214 Realistic Web Tasks for Assistant Agents](https://benchmarkingagents.com/assistantbench/): AssistantBench (Yoran et al., EMNLP 2024) tests web-assistant agents on 214 time-consuming real-world tasks. The best system, an SPA-plus-closed-book ensemble, scored 25.2%; the SeePlanAct web agent alone 11.1%; closed-book GPT-4 16.5 to 22.2%... - [AutoGen Benchmarks 2026: Multi-Agent Evals and Coverage](https://benchmarkingagents.com/autogen-benchmarks/): AutoGen is the multi-agent conversation framework from Microsoft Research. The AG2 fork, the multi-agent patterns, and where multi-agent collaboration helps and where... - [BEIR: The Zero-Shot Retrieval Benchmark for RAG (Metrics, Methodology)](https://benchmarkingagents.com/beir/): BEIR is the zero-shot information-retrieval benchmark used to evaluate the retrieval stage of RAG. What it measures, its 18 datasets across nine... - [Benchmark Contamination: How It Happens, How to Detect It](https://benchmarkingagents.com/benchmark-contamination/): Why MMLU, HumanEval, and MBPP all suffered training-data contamination, what to look for in any benchmark, the structural fixes (rolling cutoffs, private... - [The 2026 LLM Benchmark Reference: 20 Benchmarks Indexed](https://benchmarkingagents.com/benchmarks-list/): A 2026 LLM benchmark reference. 20 benchmarks across knowledge, coding, reasoning, agentic, multimodal, and human-preference categories, with what each measures, task count,... - [Browser-Agent Benchmarks 2026: WebArena, OSWorld, Visual WebArena Compared](https://benchmarkingagents.com/best-benchmarks-for-browser-agents/): WebArena vs OSWorld vs Visual WebArena vs Mind2Web vs WebShop. Which browser-agent benchmark fits which workload, what each measures, and what to... - [Coding-Agent Benchmarks 2026: Which to Use, Which to Skip](https://benchmarkingagents.com/best-benchmarks-for-coding-agents/): SWE-bench Verified, LiveCodeBench, Terminal-Bench, HumanEval, MBPP, BigCodeBench. Which coding-agent benchmark fits which workload, where each is brittle, and what to quote in... - [RAG Benchmarks 2026: BEIR, MTEB, RAGAS, MS-MARCO Compared](https://benchmarkingagents.com/best-benchmarks-for-rag/): BEIR for retrieval, MTEB for embedding, RAGAS for end-to-end RAG, HotPotQA and MultiHopRAG for multi-hop reasoning. Which RAG benchmark fits which question,... - [Reasoning Benchmarks 2026: GPQA-Diamond, ARC-AGI-2, HLE, BIG-Bench Hard](https://benchmarkingagents.com/best-benchmarks-for-reasoning/): GPQA-Diamond, ARC-AGI-2, Humanity's Last Exam, BIG-Bench Hard, MATH, AIME. Which reasoning benchmark fits which question, where each is brittle, and what to... - [Tool-Use Benchmarks 2026: BFCL, T-Eval, ToolBench, Tau-Bench Compared](https://benchmarkingagents.com/best-benchmarks-for-tool-use/): Berkeley Function-Calling Leaderboard (BFCL), T-Eval, ToolBench, Tau-Bench, ToolACE. Which tool-use benchmark fits which agent shape, and what to quote in 2026 for... - [BFCL v4: The Berkeley Function-Calling Leaderboard, Explained](https://benchmarkingagents.com/bfcl-function-calling/): Berkeley Function-Calling Leaderboard v4 (Gorilla, UC Berkeley) is the standard function-calling benchmark. Core categories (simple, parallel, multi-turn, live) plus v4 agentic evaluation:... - [BIG-Bench Hard: 23 Tasks, 6,511 Examples, Frontier Above 90%](https://benchmarkingagents.com/bigbench-hard/): BIG-Bench Hard (BBH) is the 23-task subset of BIG-Bench where 2022 LLMs fell below the average human-rater baseline. Methodology, why it still... - [BrowserGym: One Gym Interface for WebArena, VisualWebArena, MiniWoB++](https://benchmarkingagents.com/browsergym/): BrowserGym (ServiceNow Research, 2024) is a unified Gym-style API for web-agent benchmarks. Same agent code runs against WebArena, VisualWebArena, MiniWoB++, WorkArena, and... - [Chatbot Arena: Elo Ratings, Methodology, Caveats](https://benchmarkingagents.com/chatbot-arena/): Chatbot Arena (LMArena) ranks LLMs by human pairwise preference. Methodology, Bradley-Terry-Luce Elo scoring, style control, vote-volume requirements, and what the rating does... - [Cost Per Eval 2026: Running MMLU, SWE-bench, GAIA at Scale](https://benchmarkingagents.com/cost-per-eval/): What it costs to run each major benchmark at frontier-model API prices in 2026. MMLU-Pro, GPQA-Diamond, SWE-bench Verified, GAIA, OSWorld, Tau-Bench. Per-token... - [CrewAI Benchmarks 2026: Where It's Tested and Where It's Not](https://benchmarkingagents.com/crewai-benchmarks/): CrewAI is the role-and-task agent framework. Its crew patterns, the production strengths it was designed for, and why its benchmark coverage is... - [Building a Custom LLM Eval Pipeline in 2026 - End-to-End Guide](https://benchmarkingagents.com/custom-evals/): Tool-agnostic guide to building custom LLM evaluation pipelines. The 5-stage eval lifecycle, dataset sizing, metric selection, tooling decision tree, and common mistakes.... - [DSPy Benchmarks 2026: GSM8K, HotPotQA, Programmatic Optimisation](https://benchmarkingagents.com/dspy-benchmarks/): DSPy compiles structured agent pipelines and optimises prompts programmatically. What the framework is, its optimisers, and where it wins versus prompt-engineering-by-hand - [AI Benchmarking FAQ - 14 Questions About Evaluating LLMs and Agents](https://benchmarkingagents.com/faq/): Answers to 14 frequently asked questions about AI benchmarking, LLM evaluation, and agent evals. What is benchmarking? Which benchmarks matter in 2026?... - [Monitoring LLMs and Agents in Production - Online Eval, Drift, Cost-per-Correct](https://benchmarkingagents.com/for-production-monitoring/): Practical guide to LLM production monitoring. Offline vs online eval, what to monitor, sampling strategy, cost-per-correct-answer, drift detection, and tooling recommendations - [FrontierMath Benchmark: 338 Research-Grade Problems, Tiers 1-4](https://benchmarkingagents.com/frontiermath/): FrontierMath is the hardest mathematics benchmark in active use: 338 unpublished, expert-vetted problems from Epoch AI, split into a 295-problem base (Tiers... - [GAIA Benchmark: 466 Assistant Tasks Across Three Difficulty Levels](https://benchmarkingagents.com/gaia-benchmark/): GAIA tests general AI assistant capability across 466 multi-step tasks with three difficulty levels. Methodology, leveled scoring, the private test set, and... - [AI Benchmarking Glossary - 25 Terms for LLM and Agent Evaluation](https://benchmarkingagents.com/glossary/): Definitions of 25 key terms in AI benchmarking and LLM evaluation: N-shot, pass@k, LLM-as-judge, faithfulness, SOTA, saturation, contamination, CoT, and more. Clear... - [GPQA, ARC-AGI, and Humanity's Last Exam - The Reasoning Frontier 2026](https://benchmarkingagents.com/gpqa-arc-agi/): GPQA-Diamond, ARC-AGI-2, and Humanity's Last Exam are the reasoning benchmarks that still have headroom in 2026. What each measures, who built them,... - [Safety Eval Suites 2026: HarmBench, JailbreakBench, AILuminate Compared](https://benchmarkingagents.com/harm-safety-evals/): HarmBench (Mazeika et al., ICML 2024), JailbreakBench (Chao et al., 2024), and AILuminate (MLCommons, 2024) are the three canonical safety eval suites.... - [Humanity's Last Exam: 2,500 Frontier Questions Explained](https://benchmarkingagents.com/hle-humanitys-last-exam/): HLE is the frontier-difficulty replacement for saturated knowledge benchmarks. 2,500+ expert-curated questions across mathematics, sciences, humanities, where current models score well below... - [HotpotQA: The Multi-Hop QA Benchmark (Distractor + Fullwiki)](https://benchmarkingagents.com/hotpotqa/): HotpotQA is the foundational multi-hop question-answering benchmark: 113,000 Wikipedia questions that require reasoning across two documents, with sentence-level supporting facts for explainability.... - [Human vs Automated LLM Evaluation - Chatbot Arena, LMSYS, and Cost-Benefit](https://benchmarkingagents.com/human-vs-automated/): When to use human evaluation vs automated LLM-as-judge. How Chatbot Arena works, the cost of human evaluation, and the hybrid approach most... - [HumanEval, MBPP, and LiveCodeBench - Code Benchmarks in 2026](https://benchmarkingagents.com/humaneval/): HumanEval is saturated in 2026. Learn why LiveCodeBench is now the standard for code generation benchmarking, how contamination is prevented, and what... - [Inspect by UK AISI: The Eval Framework Behind Frontier Safety Tests](https://benchmarkingagents.com/inspect-uk-aisi/): Inspect (UK AI Safety Institute, 2024) is the open-source eval framework UK AISI uses to test frontier models. Python, MIT-licensed, integrates with... - [LangGraph Benchmarks 2026: Where It Scores, Where It Doesn't](https://benchmarkingagents.com/langgraph-benchmarks/): LangGraph is the LangChain agent framework for stateful multi-step agent graphs. What benchmarks the framework targets, how to read a LangGraph benchmark... - [LegalBench: 162 Legal Reasoning Tasks, GPT-4 Macro-F1 at 77.0](https://benchmarkingagents.com/legalbench/): LegalBench (Guha et al., Stanford CRFM 2023) is the canonical legal-reasoning benchmark. 162 tasks across IRAC, doctrine application, and rule conclusion. Methodology,... - [LiveCodeBench 2026: Anti-Contamination Code Eval Explained](https://benchmarkingagents.com/livecodebench/): LiveCodeBench solves the HumanEval contamination problem with a rolling-cutoff design: only problems released after the model's training cutoff are scored. Methodology, scoring,... - [LLM-as-Judge 2026 - Methodology, Prompts, Bias, When to Use It](https://benchmarkingagents.com/llm-as-judge/): Neutral methodology guide to LLM-as-Judge evaluation. Single-point scoring vs pairwise comparison, G-Eval, known bias vectors (position, verbosity, self-preference), and template prompts you... - [MATH Benchmark: 12,500 Problems, 99%+ Saturation, What Replaced It](https://benchmarkingagents.com/math-benchmark/): MATH (Hendrycks et al., 2021) is a 12,500-problem competition math benchmark. Frontier accuracy crossed 99% in 2025, ending its usefulness for top-tier... - [MedQA: 12,723 USMLE Questions, Med-PaLM 2 at 86.5% Accuracy](https://benchmarkingagents.com/medqa-medical-benchmark/): MedQA (Jin et al., 2020) is the USMLE-style medical reasoning benchmark. Plus MultiMedQA: 7 medical question-answering datasets combined by Google for the... - [Methodology: How benchmarkingagents.com Verifies AI Benchmark Scores](https://benchmarkingagents.com/methodology/): Source URLs per benchmark, the six-question rubric, what is in scope, what is out of scope, saturation and contamination tagging, refresh cadence,... - [Mind2Web: 2,350 Web Tasks Across 137 Sites, Offline Action Prediction](https://benchmarkingagents.com/mind2web/): Mind2Web (Deng et al., NeurIPS 2023) is the canonical offline web-action-prediction benchmark. 2,350 tasks across 137 real websites; GPT-4 with MindAct scored... - [MMLU, MMLU-Pro, and MMMU Explained: The Knowledge Benchmarks in 2026](https://benchmarkingagents.com/mmlu/): A complete guide to MMLU, MMLU-Pro, and MMMU. Why MMLU is saturated in 2026, when to use MMLU-Pro, and how MMMU extends... - [MMLU-Pro 2026: Why It Replaced MMLU and Current SOTA](https://benchmarkingagents.com/mmlu-pro/): MMLU-Pro replaced saturated MMLU with 10-choice questions across 14 categories. Methodology, SOTA progression 2024 to 2026, and how it differs from the... - [MMMU Benchmark: 11,500 Multimodal Questions, Frontier Above 70%](https://benchmarkingagents.com/mmmu-benchmark/): MMMU (Yue et al., CVPR 2024) is the canonical college-exam-style multimodal reasoning benchmark. 11,500 questions across 30 subjects, 6 disciplines. Methodology, MMMU... - [Frontier Models on Benchmarks 2026: Why We Don't Reprint a Score Matrix](https://benchmarkingagents.com/model-on-benchmark/): How to read frontier-model benchmark scores honestly, and why a single per-model score matrix is unreliable: cross-paper numbers aren't comparable, vendor cards... - [MS MARCO: The Passage-Ranking Benchmark (v1 and v2) Explained](https://benchmarkingagents.com/ms-marco/): MS MARCO is the canonical large-scale passage-ranking benchmark: 8.8 million passages, ~500K real Bing queries, scored with MRR@10. The three tasks (QA,... - [MTEB: The Massive Text Embedding Benchmark (Tasks, Metrics, MMTEB)](https://benchmarkingagents.com/mteb/): MTEB is the standard benchmark for selecting embedding models. What it measures, its eight task types and per-task metrics, how the 2025... - [OpenAI Agents SDK Benchmarks 2026: Coverage and Caveats](https://benchmarkingagents.com/openai-agents-sdk-benchmarks/): The OpenAI Agents SDK is OpenAI's first-party agent framework. What it is, how the Responses API and hosted tools integrate with the... - [OpenAI Evals: 400+ Public Evals, How to Add One in 30 Lines](https://benchmarkingagents.com/openai-evals-framework/): OpenAI Evals (released March 2023) is the open-source evaluation framework from OpenAI. 400+ public evals contributed by the community. YAML-driven, supports both... - [OSWorld Benchmark: Computer-Use Agents on Real Operating Systems](https://benchmarkingagents.com/osworld/): OSWorld-Verified top score 90.19% (Intelligence-Indeed Agent, official results read 18 Nov 2026), above the paper's 72.36% human baseline. 369 real-OS tasks, scoring, task areas and harness sensitivity. - [pass@1 vs pass@k: Why HumanEval Numbers Are Often Misquoted](https://benchmarkingagents.com/pass-at-k/): pass@k (Chen et al., HumanEval paper, 2021) is an unbiased estimator over k samples. pass@1 is greedy. pass^k is consistency. The differences... - [Prompt Template Variance: Same Model, Same Benchmark, 10+ Point Swing](https://benchmarkingagents.com/prompt-template-variance/): Sclar et al. (ICLR 2024) showed prompt-template variance moves LLM benchmark scores by up to 76 points on the same model. Methodology,... - [RAG Evaluation 2026 - Faithfulness, Relevancy, and Context Metrics](https://benchmarkingagents.com/rag-eval/): Neutral, tool-agnostic guide to evaluating RAG pipelines. Faithfulness, answer relevancy, context precision, and context recall explained. How to build a golden dataset... - [RAGAS: Automated RAG Evaluation (Metrics, Reference-Free Scoring, 2026)](https://benchmarkingagents.com/ragas/): RAGAS is the standard open-source framework for scoring RAG pipelines. What it measures, its core metrics (faithfulness, response relevancy, context precision, context... - [RepoBench: 27,000 Multi-File Code Completion Tasks, GPT-4 at 36.4%](https://benchmarkingagents.com/repobench/): RepoBench (Liu et al., ICLR 2024) tests code completion across multiple files in real repositories. Methodology, retrieval-first design, the launch baseline, and... - [Why Benchmark Scores Fail to Reproduce: 5 Causes, Receipts From Papers](https://benchmarkingagents.com/reproducibility-evals/): Burnell et al. (2024) and Biderman et al. (2024) documented systematic reproducibility failures across MMLU, HumanEval, ARC, and SWE-bench. Five root causes,... - [Stanford HELM: 7 Metrics, 30 Scenarios, Why Coverage Beats Headline Score](https://benchmarkingagents.com/stanford-helm/): HELM (Holistic Evaluation of Language Models, Stanford CRFM, 2022) evaluates LLMs on 7 metrics across 30+ scenarios. Methodology, why the holistic framing... - [SWE-bench Verified Explained: 2026 Methodology, Tiers, Caveats](https://benchmarkingagents.com/swe-bench/): A complete guide to SWE-bench and SWE-bench Verified. Methodology breakdown, SOTA progression, and honest limitations. The canonical coding-agent benchmark explained - [Tau-Bench 2026: Customer-Service Agent Eval, Methodology, Tiers](https://benchmarkingagents.com/tau-bench/): Tau-Bench evaluates customer-service agents on realistic retail and airline tasks with user simulation, tool use, and policy adherence. Methodology, scoring, harness sensitivity,... - [Tau-Bench Retail and Airline: 165 Tasks, Frontier Pass^1 Below 70%](https://benchmarkingagents.com/tau-bench-retail-airline/): Tau-Bench (Sierra Research, 2024) retail and airline domains. 165 tasks (115 retail, 50 airline), pass^k metric for consistency, policy-following requirements. Methodology and... - [Tau2-Bench (τ²-bench): Dual-Control Agent Eval and the Telecom Domain](https://benchmarkingagents.com/tau2-bench/): τ²-bench is Sierra's dual-control successor to tau-bench: both the agent and the simulated user can act on shared state, formalised as a... - [Tau3-Bench (τ³-bench) 2026: What It Is, Domains, and How It Differs from Tau-Bench](https://benchmarkingagents.com/tau3-bench/): τ³-bench is the third generation of Sierra's tool-agent-user benchmark: multimodal, knowledge-aware customer-service evaluation across retail, airline, telecom, and a new banking-knowledge domain,... - [Terminal-Bench: Shell Agent Benchmark, Methodology and Tiers](https://benchmarkingagents.com/terminal-bench/): Terminal-Bench evaluates shell agents on containerised real Linux tasks. Methodology, container reproducibility, task taxonomy, three difficulty tiers, and harness sensitivity - [LLM Eval Tools Compared 2026: Braintrust, Langfuse, Promptfoo, DeepEval, Galileo + Alternatives](https://benchmarkingagents.com/tools-compared/): Neutral side-by-side comparison of 9 LLM eval tools in 2026: Braintrust, Langfuse, LangSmith, Arize Phoenix, Promptfoo, Galileo, HoneyHive, W&B Weave, DeepEval. Head-to-head... - [WebArena: Real Web-Task Agent Benchmark, Methodology and Caveats](https://benchmarkingagents.com/webarena/): WebArena measures browser agents on real-site task completion across e-commerce, forums, GitLab, and a CMS. Methodology, scoring, the four self-hosted sites, and... - [What LLM Benchmarks Don't Measure - Contamination, Saturation, Blind Spots](https://benchmarkingagents.com/what-these-benchmarks-miss/): Editorial: Five honest problems with the AI benchmark landscape in 2026. Contamination, saturation, self-reporting conflicts, methodology gaming, and the real-world gap. A... --- *Generated 2026-08-26 from AI Benchmarks 2026: Which Benchmark to Use and How to Read It site source.*