What is AutomationBench?

Zapier’s AutomationBench is an open benchmark, released April 20, 2026, that tests whether an AI agent can finish a cross-app business workflow: it drops the agent into a simulated company of 47 apps and about 500 API endpoints, hands it one request, and passes the task only if the data it left behind is exactly right.

Tasks cover Sales, Marketing, Operations, Support, Finance, and HR, generated synthetically from workflow patterns on Zapier’s platform, per the whitepaper by Daniel Shepard and Robin Salimans (arXiv:2604.18934). The agent works alone with two tools, a keyword search over API schemas and a request executor, so finding the right endpoint is part of the job. Environments hide traps: near-duplicate records, stale spreadsheet rows, and policies buried in an inbox.

Grading ignores the agent’s reply and uses no LLM judge: fixed assertions check the final state of the apps. Sierra’s tau-bench grades end state too, but the whitepaper notes that each task in its latest version, tau3-bench, still stays inside one application.