Independent reference. No vendor affiliation. Reprinted scores dated to their source; elsewhere we link the official board. Editorial policy.
Vol. III · Apr 2026
THE

Benchmarking Agents Review

An independent reference on AI evaluation
ISSN 2026-0428
Abstract
What214 realistic web-browsing tasks, 5 to 90 minutes of human effort each.
WhoYoran, Amouyal, Malaviya, Bogin, Press, Berant (Tel Aviv University, EMNLP 2024).
BaselinesBest system (SPA plus closed-book ensemble) 25.2%; SeePlanAct web agent 11.1%; closed-book GPT-4 16.5 to 22.2% in the launch paper; far from saturated.
Projectassistantbench.github.io
Section II.v Agent Benchmarks|Reviewed 2026

AssistantBench: 214 Realistic Web Tasks for Assistant Agents

The benchmark designed around what users actually ask assistants to do: multi-step web research with a verifiable answer.

I

Construction

The Tel Aviv University team built AssistantBench by sourcing tasks from three streams: (1) MTurk worker submissions of recent tasks they actually wanted help with, (2) university administrator workflows (course catalogue lookups, scholarship eligibility checks), and (3) author-curated tasks designed to stress specific failure modes. Each task ships with a gold answer extracted by humans and a verification protocol.

The benchmark targets tasks that require live web research rather than recall. Closed-book GPT-4 with browsing disabled still reaches 16.5% with an instruction prompt and 22.2% with a one-shot example, higher than any pure web agent, but the paper shows the figure is misleading: 85% of those closed-book errors are hallucinated answers, so the model scores on a high answer rate rather than genuine knowledge of the web.

II

Reading the scores

The launch paper's anchors are clear. The best result, 25.2%, comes from an ensemble that runs the SeePlanAct (SPA) web agent and falls back to a closed-book model when the agent abstains. SPA alone scores 11.1%, the earlier SeeAct web agent only 4.1%, and browsing-disabled closed-book GPT-4 reaches 16.5 to 22.2% (higher than the pure web agents, but mostly by hallucinating). No system passes 26%. We do not reprint a per-model table of current scores, because a responsibly-sourced one cannot be published here: there is no continuously-maintained board we re-verify, current numbers come from vendor reports under different browser harnesses, and the live-web design means a task's gold answer can change between runs, so 2024 and 2026 numbers are not strictly comparable.

For current results, read the official AssistantBench project page and each submission's harness together. To choose the right benchmark for your use case rather than chase a single figure, start from the homepage task picker.

III

Limitations

Live-web evaluation has a reproducibility cost. A 2024 task that asked for the rent of a specific listing has a different gold answer in 2026 because the listing changed. The AssistantBench team partly addresses this by snapshotting page contents at task creation time, but agents in the live browser see the current page, not the snapshot. This means scores from 2024 papers and 2026 papers are not strictly comparable for time-sensitive tasks.

GAIA: the other realistic assistant benchmark →Mind2Web for action prediction →Browser-agent benchmarks compared →
Reader Questions
Q.01What does AssistantBench test?+
AssistantBench is 214 web-browsing tasks designed to take a human between 5 and 90 minutes. Examples include 'find the average price of a 2-bedroom rental in three named neighbourhoods' or 'list the speakers at the most recent NeurIPS keynote and their affiliations'. Tasks require multi-page navigation, cross-source synthesis, and structured-output extraction.
Q.02What is the headline score?+
In the launch paper the best result is 25.2% accuracy, from an ensemble that runs the SeePlanAct (SPA) web agent and falls back to a closed-book model when the agent abstains. SPA on its own scores 11.1%, the earlier SeeAct web agent 4.1%, and closed-book browsing-disabled GPT-4 reaches 16.5% (instruction prompt) to 22.2% (one-shot prompt). No system exceeds 26%. Those are the cleanly-sourced anchors. We do not reprint a per-model table of current scores: there is no continuously-maintained board we re-verify, vendor numbers use different browser harnesses, and live-web drift means a 2024 task and its 2026 re-run are not strictly comparable.
Q.03How does AssistantBench score answers?+
Answers are short structured outputs (numbers, lists, JSON-like records) and scoring is fully automatic with partial credit: token-level F1 for strings, a magnitude-tolerant score for numbers, and key-value F1 for dictionary records. A task is only included if it has a closed-form answer that can be automatically verified and is unlikely to change quickly, so free-text summaries and subjective judgements are filtered out at collection time rather than graded by an LLM judge.
Q.04Why is GPT-4 stuck below 30%?+
The paper's manual error analysis separates two failure profiles. When the SeePlanAct web agent fails, the dominant causes are navigation errors (36.7% of its errors, where it never reaches the page holding the answer) and grounding failures (23.3%, where it misreads the page it does reach), with the rest split between wrong answers (20.0%), technical or parsing issues (16.7%), and giving no answer (3.3%). Closed-book models fail differently: 85% of their errors are hallucinated answers and 15% are outdated. So the sub-30% ceiling is open-web navigation difficulty for agents and fabrication for memory-only models, not one single capability wall.
Q.05How is this different from WebArena and Mind2Web?+
WebArena uses controlled simulated environments (a self-hosted Reddit-like forum, a self-hosted GitLab). Mind2Web uses static page snapshots with offline action prediction. AssistantBench uses the live public web. The cost is that the web changes under the agent and reproducibility weakens over time; the benefit is that AssistantBench measures what users actually want.

Sources

  1. [1] Yoran et al. (2024): arxiv.org/abs/2407.15711
  2. [2] AssistantBench project: assistantbench.github.io
From the editor

Benchmarking Agents Review is published by Digital Signet, an independent firm that builds and ships AI agents in production for mid-market companies. If you are evaluating, designing, or productionising LLM agents and want a working second opinion, get in touch.

Book a 30-min scoping callDigital Signet →

30 minutes, free, independent.·1-page action plan within 48h.·Honest if not the right fit.

Correspondence

Didn't find your answer?

Ask us. A real person reads every question and we answer the ones we can, with sources. If your question would help other readers, we may publish an anonymised version, with your permission. General reference only.

Ask a question

questions@benchmarkingagents.com