NO PER-MODEL SCORE TABLE — this page explains construction, methodology, and limitations only. Agent-benchmark numbers move weekly and the official leaderboards are JavaScript-rendered and lag the current frontier, so we do not reprint a score table we cannot stand behind. Each “official leaderboard” link below goes to the live source for current per-model numbers.
AI Agent Benchmarks - SWE-bench, WebArena, AgentBench, Terminal-Bench, OSWorld, Tau-Bench
The six benchmarks · tap through to each official leaderboard for live scores
We do not reprint per-model scores: the official boards lag the current frontier and vendor-reported numbers come from harnesses we cannot inspect. Each link goes to the live source.
Agent benchmarks measure something fundamentally different from LLM benchmarks. The question is not "can the model answer this question?" but "can the model complete this multi-step task that requires tools, state management, and error recovery?" The distinction matters because high scores on MMLU, GPQA, and HumanEval do not predict agentic capability.
A model that scores very high on MMLU-Pro can still fail at a multi-step WebArena task that requires synthesising information across several pages, entering it correctly into a form, and verifying the result. Agentic success requires planning, tool-use discipline, error detection, and recovery: capabilities that static question-answering benchmarks do not stress. Reported scores on the major agent benchmarks sit well below the levels seen on knowledge benchmarks (OSWorld in particular remains a hard, low-scoring open problem), which is the honest signal that these are unsolved tasks. For current per-model numbers, read each benchmark's official leaderboard rather than any figure reprinted second-hand.
SWE-bench Verified
Construction
500 human-verified tasks drawn from real GitHub issues in 12 Python repositories (Django, SymPy, Astropy, matplotlib, and others). Each task: given the repo state, the issue, and a failing test, the agent must produce a passing patch.
Methodology
Agent has access to the repository, the issue description, and a test runner. It produces a patch. The patch is applied and the test suite runs. Success: all fail_to_pass tests pass AND all pass_to_pass tests still pass.
Strengths
Grounded in real engineering tasks. Pass/fail objective. Verified subset eliminates ambiguous tasks.
Known Limitations
Python-only. Test-flakiness exists. Solutions in git history create contamination risk. A patch that passes tests is not always production-ready.
WebArena
Construction
812 tasks across 5 fully functional web applications: Reddit clone (Postmill), e-commerce (OneStopShop), collaborative coding (GitLab), maps (OpenStreetMap), and a content management system. All applications run in a sandboxed environment.
Methodology
Agent uses a browser. Tasks range from navigation (find the top-voted post this week) to transaction (buy a specific product) to information extraction. Success is evaluated programmatically via page state or API calls.
Strengths
Multi-application, realistic web tasks. Functional applications not screenshots. Task variety.
Known Limitations
Web applications are static snapshots - real web is dynamic. Some tasks have multiple valid paths not all captured. Domain limited to the 5 provided applications.
AgentBench
Construction
1,091 tasks across 8 distinct agent environments: OS shell interaction, database management (MySQL/PostgreSQL), knowledge graph querying, digit card game, 2048, House-Holding (household tasks), web browsing, and web shopping.
Methodology
Each environment has its own success criterion. OS tasks: did the command achieve the goal? Database: did the query return the correct result? Games: did the agent win? Shopping: did the agent purchase the correct item at the best price?
Strengths
Broad environmental coverage. Tests different agent capabilities. OS and database tasks highly relevant for engineering agents.
Known Limitations
8 environments is still a narrow slice of real-world tasks. Game tasks (2048, card games) may not predict useful agentic capability. Dataset may have contamination in more common tasks.
Terminal-Bench
Construction
Terminal/shell-based tasks inside isolated Docker containers: scripting, system administration, debugging, data manipulation, security, and CI/CD. A compact, freshly-written corpus (80 tasks at the May 2025 v1 launch; 89 in the v2 set) rather than a large derived one.
Methodology
Agent has access to a bash terminal in a containerized environment. Each task ships a per-task verification script (run-tests.sh) that checks the container's final state - file contents, process list, config, command output - against the expected outcome.
Strengths
Highly relevant for SRE and DevOps use cases. Terminal tasks are atomic and objectively evaluable. Container isolation means reproducibility.
Known Limitations
Shell tasks vary widely in ambiguity. Some tasks have multiple correct solutions. Less published academic coverage than SWE-bench or WebArena.
OSWorld
Construction
369 tasks on a real Ubuntu Linux desktop (with Windows and macOS variants in extension sets) using real applications: web browsers (Chrome, Firefox), productivity software (LibreOffice Writer, Calc, GIMP, VLC), IDEs (VS Code), and system tools.
Methodology
Agent sees the screen (screenshot or accessibility tree). It issues actions: clicks, keystrokes, scroll. Task completion evaluated by checking the application state against a ground-truth specification.
Strengths
Covers real computer use. Not web-confined. Tests cross-application workflows. Evaluation is objective.
Known Limitations
Virtual machine infrastructure overhead. Applications and task definitions are static - real desktop environments are more dynamic. Lower SOTA than other benchmarks suggests the field is early here.
Tau-Bench
Construction
165 customer-service tasks across two domains: retail (115 tasks - returns, exchanges, account changes) and airline (50 tasks - cancellations, rebookings, baggage). Each task pairs a simulated user with a domain database and typed API tools; the agent must resolve the request over multiple turns while obeying the domain's written policy.
Methodology
A trajectory succeeds only when the user's intent is satisfied, the database ends in the correct state, and no action violated policy. The headline metric is pass^k (consistency across k independent runs) with the user simulator fixed to GPT-4o.
Strengths
Policy compliance scored as a first-class metric. Customer-service tasks have clear commercial relevance. Multi-turn dialogue with a simulated user is realistic.
Known Limitations
Only two domains (retail, airline). Original board frozen at the late-2024 model set; newer models run on the successor tau2/tau3-bench. Simulator choice affects scores.
Comparison Table
| Benchmark | Domain | Tasks | Success Criterion | Leak Risk |
|---|---|---|---|---|
| SWE-bench Verified | Software Engineering | 500 | Tests pass + no regression | Medium (public git history) |
| WebArena | Web Navigation | 812 | Page state / API check | Low (sandboxed apps) |
| AgentBench | Multi-environment | 1,091 | Per-environment check | Low-Medium |
| Terminal-Bench | Shell / DevOps | 89 | File system state | Low (Docker isolated) |
| OSWorld | Computer Use | 369 | App state check | Low (VM isolated) |
| Tau-Bench | Customer-service dialogue | 165 | Resolution + policy compliance | Medium |
| Construction and methodology summary. Per-model scores are not reprinted here; read each benchmark's official leaderboard (linked above) for current numbers. | ||||
The Agent Benchmark Methodology Crisis
Agent benchmarks are less stable than LLM benchmarks for four structural reasons. First, environment drift: web pages change, APIs evolve, and applications update - the sandboxed environment in WebArena 2023 is not the same as the web environment a deployed agent faces in 2026. Benchmark environments are frozen; real environments are not.
Second, test flakiness: agentic tasks often have non-deterministic environments. A task that passes 70% of the time in isolation might pass 90% of the time with one random seed and 50% with another. Benchmark scores aggregate over this variance but rarely report confidence intervals.
Third, solution leakage: SWE-bench tasks come from public GitHub issues whose solutions are in the same repository's git history. Any model trained after the tasks were created may have seen the solutions. The Verified subset partially addresses this, but the risk is structural.
Fourth, and most importantly, the benchmark-to-production gap: a 70% SWE-bench Verified score does not mean the agent can handle 70% of real software engineering work. The benchmark task distribution (12 Python repos, specific issue types) does not match the distribution of a real engineering team's backlog. A model that excels at SymPy bugs may struggle with Django authentication.
Use agent benchmarks as a floor test, not a capability certificate. High scores are necessary but not sufficient.
Frequently Asked Questions
Which agent benchmark matters most in 2026?+
How does SWE-bench Verified differ from original SWE-bench?+
Do agent benchmarks predict real-world agent performance?+
Can I run these benchmarks myself?+
What is OSWorld trying to measure?+
Continue Reading
Where to go next
SWE-bench Verified explained
Construction, contamination risks, and how to read a Verified score honestly. The most-cited agent benchmark in 2026.
What these benchmarks miss
Five structural problems with the agent-benchmark landscape: environment drift, test flakiness, solution leakage, methodology opacity, and the benchmark-to-production gap.
Eval tools compared
Nine platforms reviewed independently: Braintrust, Langfuse, LangSmith, Arize Phoenix, Promptfoo, Galileo, HoneyHive, W&B Weave, DeepEval. OSS vs cloud, self-host options, head-to-head matchups, honest avoid-if notes.
Build your own evals
How teams actually measure the workflow they ship. Custom golden datasets, LLM-as-judge, online monitoring, regression tracking. The practitioner stack, no vendor capture.
Human vs automated evaluation
When to use human evaluation vs LLM-as-judge. Cost-benefit math, where automation breaks down, hybrid approaches that production teams actually deploy.
Full benchmark reference
Every benchmark we cover: model benchmarks (MMLU-Pro, GPQA, ARC-AGI), agent benchmarks, code benchmarks, RAG benchmarks. Sourced, dated, and cross-linked.