# What does a tau-bench score mean?

A tau-bench score averages across several domains that don't play out the same way, and the single-attempt version of that score overstates how reliable an agent actually is. In tau2-bench, GPT-4.1's pass@1 runs 74% on retail and 56% on airline, then drops to 34% on telecom, a domain tau3-bench kept: three domains, one model, a 40-point spread hidden inside whatever the overall number reads.

The consistency gap compounds it. The original tau-bench paper's own numbers show GPT-4o solving 61.2% of retail tasks on a single try, pass@1, but under a metric that only counts a run if it succeeds every time, pass^8, that same setup falls under 25%. [ReliabilityBench uses the identical pass^k idea](/answers/reliabilitybench/how-do-you-measure-whether-an-agent-succeeds-consistently) for tool-using agents generally, and the gap between pass@1 and pass^8 here is how much a single clean run can overstate what a real customer actually gets.

Read one tau-bench percentage as the easiest domain and the luckiest run, not the agent's typical day.

---

Sources:
- τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (arXiv:2506.07982): https://arxiv.org/abs/2506.07982 (fetched 2026-10-04)
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045): https://arxiv.org/abs/2406.12045 (fetched 2026-10-04)

Source: https://tessary.ai/answers/tau-bench/what-does-a-tau-bench-score-mean
More on Tau bench: https://tessary.ai/answers/tau-bench
From Tessary, agent reliability for AI agents in production: https://tessary.ai