Tau bench

tau-bench is Sierra's test for customer-service agents. Its current version, tau3-bench, came out in March 2026. The AI plays a support agent for an airline, a retailer, a telecom, or a bank. A simulated customer talks to it, the agent has a policy document it must follow, and it has tools to look up and change the customer's account.

The score ignores how the conversation sounded. A task passes only if the right changes were made to the account: the refund issued, the card frozen, the dispute opened, and nothing else changed. Each task is also run several times, and the strict version of the score counts only tasks that passed every time, because a real customer gets one attempt.

Two things make it hard. Since tau2-bench, the simulated customer can act too, restarting a router or reading a code off a screen, so the agent has to guide a person rather than just answer them, and scores dropped sharply when that was added. And tau3-bench adds a banking domain where the answer is buried in 698 internal documents, plus a voice mode with interruptions, accents, and background noise.

The best frontier model completes about a quarter of the banking tasks. In the voice mode, agents that score 85% on text score 26 to 38% on the same tasks spoken aloud. If your agent talks to customers, those are the numbers to compare against, not the older text-only ones.