How is tau-bench scored?

Tau-bench checks whether the account ended up in the state a correct run would leave it in, the refund issued, the subscription canceled, and nothing else changed alongside it, and ignores how the conversation sounded getting there. A polite, coherent transcript that makes the wrong account change still fails.

The paper’s second departure from a normal benchmark is running each task more than once against the same policy and simulated customer, then reporting pass^k: the share of tasks that succeeded on every one of k tries, not pass@1’s rate for a single attempt. That matters because a real customer only gets the one attempt the agent actually gives them, not the best of several, so a headline pass@1 number can hide how often the same agent would fail on a retry. ReliabilityBench applies the identical pass^k metric to tool-using agents generally, not just one customer-service domain.