What changed in the latest version of AutomationBench?

Version 1.0.6, released July 31, 2026, fixed tasks that were unfair to the agent and made the private tasks harder to make up for it. Zapier made tasks easier to discover, for example by giving Google Sheets tasks a way to find the spreadsheet ID without Google Drive access. It loosened formatting checks that were too strict and dropped checks for side effects the prompt never specified. These fixes covered both public and private tasks. Zapier then made the private tasks somewhat harder to compensate for the bug fixes. It also updated the Verifiers framework to 0.2.0 and added prompt caching and refusal tracking to the evaluation infrastructure.

The previous release, 1.0.5 (July 16), added public baseline scores to the readme and expanded the tasks.

Because the private set changed difficulty in both directions, a score on 1.0.6 isn’t directly comparable with one on 1.0.5. That is a general reason benchmark scores move between versions.

sources

keep reading

More on this.

What does an AutomationBench score mean?An AutomationBench score is the share of Zapier's held-out tasks where an agent left every checked record and message right; one failed check fails the task.answer →What is AutomationBench?Zapier's AutomationBench tests AI agents on business workflows across 47 simulated apps, passing a task only if the data left in every app is exactly right.answer →Which kinds of business work do AI agents do worst at on AutomationBench?On Zapier's AutomationBench, the best any model scores is 39.17% in HR and 45.00% in Support, against 68.00% in Operations.answer →Everything on automationbenchThe definition, and every question answered under automationbench.concept →

Try Tessary on Cloud.

start for freeor self-host

free · 10,000 traces a month · 1 GB · 30-day retention · $10 one-time credit for triage and RCA · no credit card