Does a high tau-bench score mean an agent will work in production?

No. tau-bench’s own authors call even state-of-the-art models “far from being reliable for use in real-world settings,” and the reason sits in what’s grading the agent: a second LLM plays the customer, and that simulated customer carries its own limits. The original paper names three: the instruction it’s given can contain typos or leave out domain knowledge the agent needs, and the simulator itself can fall short on reasoning, arithmetic, or remembering earlier turns in a long conversation.

tau2-bench, the version after it, names a gap of its own: it “does not explicitly model the expert-novice gap inherent to most customer support tasks,” where, in the paper’s own words, “an expert must understand the user’s mental model and adapt explanations accordingly,” a judgment call a scripted simulator doesn’t have to make. A real caller can also get frustrated in ways that change what they say next, exactly what Tessary’s frustration classifier is built to catch, and no simulated customer in the benchmark does that.