How is AutomationBench different from tau-bench?

AutomationBench runs an agent alone against a simulated company’s apps with no one to talk to, and grades only the state it leaves behind in those apps; tau-bench puts the agent in a live conversation with a simulated customer and grades whether that conversation produced the right account changes.

That’s a difference in who the agent has to deal with, not just what it’s handed. AutomationBench’s agent gets one written request and finds its own path through CRM records, inboxes, calendars, and spreadsheets across 47 simulated apps, and every check runs against the apps’ final state with the agent’s own reply ignored entirely. tau-bench’s agent follows a policy document while a simulated customer pushes back or asks something the policy doesn’t cover, so the task isn’t settled until the dialogue itself resolves into the right change.

Access to run it yourself differs too. Sierra’s tau-bench code and tasks are public on GitHub, now carried forward as tau³-bench, so you can test your own agent against the full set. AutomationBench publishes 600 of its tasks on GitHub, but the leaderboard scores against a separate, harder private set per domain that isn’t released, so a local run won’t match it.