Independent reference. No vendor affiliation. Reprinted scores dated to their source; elsewhere we link the official board. Editorial policy.
Vol. III · Apr 2026
THE

Benchmarking Agents Review

An independent reference on AI evaluation
ISSN 2026-0428
Abstract
What2,350 task demonstrations on 137 real public websites across 31 domains; offline action prediction.
WhoDeng, Gu, Zheng, Chen, Stevens, Wang, Sun, Su (Ohio State University, NeurIPS 2023).
BaselineGPT-4 + MindAct at 11.2% step success in the launch paper; still discriminating.
Projectosu-nlp-group.github.io/Mind2Web
Section II.vii Agent Benchmarks|Reviewed 2026

Mind2Web: 2,350 Web Tasks Across 137 Sites, Offline Action Prediction

The offline counterpart to WebArena. Snapshot-based evaluation across the public web.

I

Construction

The Ohio State NLP group recruited annotators to demonstrate everyday web tasks: book a flight, find a recipe, sign up for a newsletter, navigate to a specific event page. Each demonstration captures the high-level instruction, the page DOM at each step, the screenshot, and the target action (click, type, select). The dataset spans 137 distinct websites across 31 domains (travel, shopping, services, entertainment, government, education, news).

Evaluation is offline. The agent reads the current page snapshot and predicts the next action. Three metrics: Element Accuracy (did the agent identify the right DOM element?), Operation F1 (did the agent choose the right action type?), and Step Success Rate (did the full action exactly match the gold?). Step Success is the headline.

II

Reading the scores

The original paper reported GPT-4 with the MindAct pipeline at 11.2% step success, with much higher element-grounding accuracy (around 53%) since picking the right element is easier than emitting the full exactly-correct action. Those are the cleanly-sourced anchors. We do not reprint a per-model progression table: there is no continuously-maintained public leaderboard we re-verify, later numbers come from scattered papers under different harnesses and dataset variants (the static set versus Mind2Web-Live), and exact step-match scoring makes cross-paper numbers hard to compare cell-to-cell.

For current results, read the official Mind2Web project page and each paper's harness and split together. To choose the right benchmark for your use case rather than chase a single figure, start from the homepage task picker.

III

When to Use Mind2Web

Mind2Web is the right benchmark when the question is "does this model pick the right action at each step?" without confounds from harness, browser tooling, or live web drift. It is wrong when the question is end-to-end task completion. Pair Mind2Web (step-level diagnostics) with WebArena or AssistantBench (end-to-end signal).

WebArena for end-to-end web tasks →AssistantBench for live-web research →Browser-agent benchmarks compared →
Reader Questions
Q.01What is Mind2Web?+
Mind2Web is a dataset of 2,350 task demonstrations collected on 137 real public websites spanning 31 domains. Each demonstration is a sequence of (page snapshot, target action) pairs. The benchmark task is to predict the correct next action given the page snapshot and the high-level instruction. Mind2Web is offline: the agent does not interact with a live browser, it scores against a fixed gold trajectory.
Q.02Why is the GPT-4 step-success rate so low?+
The original paper reported 11.2% step success rate for GPT-4 with the MindAct three-stage prompting pipeline. Step success means the predicted action exactly matches the gold action at that step. Real pages have hundreds of candidate elements; exact-match is harsh. The element-grounding accuracy (does the agent identify the correct DOM element?) is much higher, around 53%.
Q.03What is MindAct?+
MindAct is the agent architecture introduced alongside the Mind2Web dataset. It uses a small model to rank candidate elements on the page, then a large model to pick the action conditional on the top-k candidates. The two-stage design works around the context-length limit when scraping full HTML, and most subsequent Mind2Web papers use a variant of it.
Q.04Has Mind2Web been replaced by WebArena?+
Not replaced. Complementary. Mind2Web measures action prediction on snapshots of the live web (high realism, low controllability). WebArena measures end-to-end task completion in self-hosted environments (low realism on app diversity, high controllability and reproducibility). Both are useful and most browser-agent papers report on both.
Q.05Is the live web's drift a problem for Mind2Web?+
Less than for live-browser benchmarks. Mind2Web evaluates against static snapshots captured at collection time, so the gold answers do not change. The downside is the snapshots age: agents that learn 2023 layouts may not generalise to 2026 pages of the same sites. The Mind2Web-Live extension addresses this with periodic re-snapshotting.

Sources

  1. [1] Deng et al. (2023): arxiv.org/abs/2306.06070
  2. [2] Mind2Web project: osu-nlp-group.github.io/Mind2Web
  3. [3] SeeAct paper (2024): arxiv.org/abs/2401.01614
From the editor

Benchmarking Agents Review is published by Digital Signet, an independent firm that builds and ships AI agents in production for mid-market companies. If you are evaluating, designing, or productionising LLM agents and want a working second opinion, get in touch.

Book a 30-min scoping callDigital Signet →

30 minutes, free, independent.·1-page action plan within 48h.·Honest if not the right fit.