Automationbench

AutomationBench is Zapier's test of whether an AI agent can finish business work that spans several apps. Zapier released it in April 2026. Each task sets up a small simulated company across 47 apps, with CRM records, inbox threads, calendars, and spreadsheets, and gives the agent one request. The agent works alone and finds the right API calls itself.

The score counts only what the agent left behind. A task passes when every fixed check on the apps' final state holds, including checks that nothing went to the wrong people. The agent's own reply counts for nothing, and no LLM judge is used. When it launched, every frontier model scored under 10%. As of October 2026, on version 1.0.6, the top score is about 51%.

Its strength is that it grades outcomes, not claims, and its tasks hide the traps real work has: near-duplicate records, stale spreadsheet rows, and rules buried in an email. Its weakness is that the apps are simulations and the tasks are synthetic, so the authors name a risk of unrealistic or impossible tasks. The 600 public tasks are on GitHub. The leaderboard uses a harder private set that Zapier sometimes makes harder between versions, so scores don't compare across versions.