What does a tau-bench score mean?

A tau-bench score averages across several domains that don’t play out the same way, and the single-attempt version of that score overstates how reliable an agent actually is. In tau2-bench, GPT-4.1’s pass@1 runs 74% on retail and 56% on airline, then drops to 34% on telecom, a domain tau3-bench kept: three domains, one model, a 40-point spread hidden inside whatever the overall number reads.

The consistency gap compounds it. The original tau-bench paper’s own numbers show GPT-4o solving 61.2% of retail tasks on a single try, pass@1, but under a metric that only counts a run if it succeeds every time, pass^8, that same setup falls under 25%. ReliabilityBench uses the identical pass^k idea for tool-using agents generally, and the gap between pass@1 and pass^8 here is how much a single clean run can overstate what a real customer actually gets.

Read one tau-bench percentage as the easiest domain and the luckiest run, not the agent’s typical day.