AndroidWorld Benchmark: 116 Mobile UI Tasks Across 20 Android Apps

Mobile-UI agency is the frontier the web benchmarks do not measure.

The DeepMind team selected 20 Android apps representative of consumer use (Calendar, Camera, Chrome, Clock, Contacts, Files, Markor, Messages, OsmAnd, Recorder, Retro Music, Simple Calendar Pro, Simple Draw Pro, Simple Gallery Pro, Simple SMS Messenger, Tasks, Vlc, Wikipedia, Browser, OpenTracks) and authored 116 tasks across them. Each task is parameterised: a "set an alarm for 7am tomorrow" task instantiates with a randomly chosen time and label per run, which reduces memorisation across runs.

The agent receives screenshots and the Android accessibility tree at each step. It returns a touch or text action. The framework executes the action, captures the new state, and continues. Scoring is binary success per task.

The original paper reported a 30.6% success rate for its strongest baseline agent, with weaker vision-language baselines far behind. That is the one cleanly-sourced anchor; it establishes AndroidWorld as hard and far from saturated. We do not reprint a per-model progression table, because a responsibly-sourced one cannot be published here: there is no continuously-maintained public leaderboard we re-verify, scores swing heavily on the grounding harness (Set-of-Mark, ScreenAI, and similar methods move the number more than the underlying language model does), and a full 116-task emulator run is expensive enough that reproductions are scarce.

For current results, read the official AndroidWorld project page and the harness each submission used together. To choose the right benchmark for your use case rather than chase a single figure, start from the homepage task picker.

The accessibility tree on Android is noisier than the DOM. Custom views, theming, and Compose-based UIs frequently omit semantic labels, leaving the agent to ground actions visually. Vision-language grounding on mobile screens is still the limiting factor, not language understanding. This is why headline scores climb fastest when better grounding methods (Set-of-Mark, ScreenAI, UI-Net) ship rather than when language models get better.