Tau3-Bench: The Knowledge-Aware, Multimodal Successor to Tau-Bench

τ³-bench is the current generation of Sierra's tool-agent-user benchmark. It keeps the retail and airline customer-service tasks that made τ-bench the reference for dialogue agents, then extends the line to a knowledge-retrieval banking domain, full-duplex voice, and a corrected task set. If you are choosing between τ-bench, τ²-bench, and τ³-bench, this page is the map.

τ³-bench (tau3-bench) is the third major version of τ-bench, the customer-service agent benchmark Sierra released in 2024. Its README summarises the arc in one line: from text-only to multimodal, knowledge-aware agent evaluation. In practice that means τ³-bench still asks the original question, can an agent resolve a multi-turn customer request using typed API tools while obeying a written policy, but broadens the surface it tests: more domains, a retrieval-grounded knowledge task, and a voice mode.

It is not a separate project. τ³-bench is maintained inside the sierra-research/tau2-bench repository, which now identifies itself as τ³-bench and carries the earlier τ² work with it. The τ³ additions were developed on a dev/tau3 branch and merged into the maintained tree. Running it needs Python 3.12 or newer, and the project has moved to the uv package manager; the banking knowledge domain requires an extra install for its retrieval pipeline.

The τ line has three generations, and each one adds capability rather than replacing the last. The original τ-bench introduced the user simulator, policy-adherence scoring, and the pass^k consistency metric across retail and airline. τ²-bench (2025) turned it into a dual-control environment, where the simulated user can also act on shared state rather than only talking, and added a telecom domain. τ³-bench builds on that base with three concrete additions:

τ³-bench spans five task families: mock (a minimal reference domain used for smoke-testing a harness), retail, airline, telecom, and banking_knowledge. Retail and airline are the original τ-bench domains, still the closest public analogue to an e-commerce returns desk and an airline reservation system. Telecom arrived with τ²-bench and stresses the dual-control setting, where the user can take actions (toggling a setting, reading a code aloud) that the agent depends on. The banking_knowledge domain is the headline τ³ addition and is covered next. The text domains ship with the core install; banking_knowledge needs the retrieval extra.

The knowledge domain (τ-Banking in the launch paper, arXiv:2603.04370) is what makes τ³-bench a knowledge-aware benchmark rather than a pure tool-use one. It models a fintech support desk where the agent must ground its answers in a large unstructured corpus, roughly 700 interconnected knowledge documents, while still executing tool-mediated account updates. The framework abstracts knowledge access as interaction with a natural-language corpus, which lets it evaluate retrieval-based, long-context, and tool-augmented approaches inside one setting rather than hard-coding a single RAG design.

It is deliberately hard. The paper reports that even frontier models with high reasoning budgets reach only about a 25.5% pass rate, and that reliability degrades sharply over repeated trials: agents struggle both to retrieve the correct documents from a densely interlinked knowledge base and to reason accurately over the internal policies those documents describe. For teams building retrieval-grounded support agents, this is a more faithful stress test than a clean-context RAG eval, because the difficulty comes from the corpus structure and policy reasoning, not from a toy question set. See our RAG evaluation reference for how knowledge-grounding is measured more generally.

The most common mistake with the τ line is comparing a number from one version against a number from another. The original τ-bench board is frozen at the late-2024 model set (Claude 3.5 Sonnet tops it at 69.2 percent retail and 46.0 percent airline pass^1), and the repository directs new evaluation to the τ² and τ³ task sets, which fix known task bugs and add domains. Because the task set, domains, and scoring changed, a τ³-bench percentage is not comparable to an original τ-bench percentage, and neither is comparable to a vendor self-report that does not say which version and simulator it used.

For that reason this page does not reprint a per-model τ³-bench score table. Those numbers move as models and harnesses change, and the honest source is the one Sierra maintains. Read the current standings on the official repository and the source papers, and when you quote a figure, record the version, the domain (retail, airline, telecom, or knowledge), and the simulator, because all three change the number.

Reach for τ³-bench when your agent does multi-turn, tool-using customer service and you want the current, actively-maintained version of the line, particularly if you need the retrieval-grounded knowledge domain or the voice mode. If you only need the classic retail and airline scoring and want a stable target, those domains still live inside τ³-bench. For single-turn function calling, BFCL is the closer benchmark; for browser agents, WebArena; for general computer use, OSWorld; for assistant-style research, GAIA. See our tool-use benchmark comparison for how the τ line sits alongside them.