AndroidWorld Benchmark: 116 Mobile UI Tasks Across 20 Android Apps

Mobile-UI agency is the frontier the web benchmarks do not measure.

The DeepMind team selected 20 Android apps representative of consumer use (Calendar, Camera, Chrome, Clock, Contacts, Files, Markor, Messages, OsmAnd, Recorder, Retro Music, Simple Calendar Pro, Simple Draw Pro, Simple Gallery Pro, Simple SMS Messenger, Tasks, Vlc, Wikipedia, Browser, OpenTracks) and authored 116 tasks across them. Each task is parameterised: a "set an alarm for 7am tomorrow" task instantiates with a randomly chosen time and label per run, which reduces memorisation across runs.

The parameterisation is the design's load-bearing idea. The paper describes the 116 task templates as "parameterized and expressed in natural language in unlimited ways," so the fixed template count expands into an effectively unbounded pool of concrete task instances. That is a stronger contamination defence than a static task list: a model cannot have memorised the exact instance it is scored on, because the target values are drawn fresh each run and checked programmatically against the resulting app state.

The agent receives screenshots and the Android accessibility tree at each step. It returns a touch or text action (tap, swipe, type, navigate). The framework executes the action, captures the new state, and continues. Scoring is binary success per task, verified by inspecting the app or system state at the end of the trajectory rather than by matching the action sequence, so an agent that reaches the goal by an unexpected route still scores a success.

The original paper reported a 30.6% success rate for its strongest baseline agent, with weaker vision-language baselines far behind. That is the one cleanly-sourced anchor; it establishes AndroidWorld as hard and far from saturated. We do not reprint a per-model progression table, because a responsibly-sourced one cannot be published here: there is no continuously-maintained public leaderboard we re-verify, scores swing heavily on the grounding harness (Set-of-Mark, ScreenAI, and similar methods move the number more than the underlying language model does), and a full 116-task emulator run is expensive enough that reproductions are scarce.

For current results, read the official AndroidWorld project page and the harness each submission used together. To choose the right benchmark for your use case rather than chase a single figure, start from the homepage task picker.

The accessibility tree on Android is noisier than the DOM. Custom views, theming, and Compose-based UIs frequently omit semantic labels, leaving the agent to ground actions visually. Vision-language grounding on mobile screens is still the limiting factor, not language understanding. This is why headline scores climb fastest when better grounding methods (Set-of-Mark, ScreenAI, UI-Net) ship rather than when language models get better.

Use AndroidWorld when the agent you are building acts on a phone: a mobile assistant, an accessibility agent, an on-device automation tool, or an RPA-style workflow that drives real apps through the UI rather than an API. It is the canonical public measure of that capability, and the 30.6% best-baseline anchor shows how much headroom the surface still has. It is the mobile complement to the web and desktop benchmarks: WebArena for DOM-based web agents, OSWorld for whole-desktop computer use, and AndroidWorld for mobile UI. For a side-by-side of the interactive-environment family, see the browser and computer-use benchmark comparison.

Do not use AndroidWorld as a general capability score. A strong AndroidWorld number says an agent can ground and act on real mobile apps; it says nothing about reasoning depth, tool-use dialogue, or engineering ability, which belong to GAIA, Tau-Bench, and SWE-bench Verified respectively. And because the score is dominated by the grounding harness, treat any figure without its harness disclosed as uninterpretable.