# What does an AutomationBench score mean?

A score on Zapier's AutomationBench is the percentage of held-out business tasks in which an agent left a simulated company's apps in exactly the right state. Grading is strict pass/fail: every check on the final records and messages has to hold, including checks that nothing went to the wrong people, and the agent's own reply is ignored.

As of October 18, 2026, the top score on Zapier's version 1.0.6 leaderboard is 51.29%, from Google's Gemini 4 Argon (High), so the best entry still fails just under half its tasks. A task passing five of six checks earns 0.83 on Zapier's diagnostic `partial_credit` but counts as a failure in the headline.

The number also belongs to a setup, not only a model. On that leaderboard Anthropic's Claude Opus 5.5 scores 42.47% at Max effort with default fallbacks, which send a refused step to the provider's fallback model, while Anthropic's September 22, 2026 launch post reports 40.0% from Zapier's early-access runs with no fallback, so a safeguard intervention counted as a failure. Routing like that is part of the system around the model, which is [why the same model scores differently in different harnesses](/answers/agent-harness/why-does-the-same-model-score-differently-in-different-harnesses).

Compare scores only within one version. Per the project's README, the leaderboard's private set is purposely harder than the public 600-task set, and private tasks sometimes get harder in version updates, so a local run won't match it.

---

Sources:
- AutomationBench leaderboard and methodology, version 1.0.6 (Zapier): https://zapier.com/benchmarks (fetched 2026-10-18)
- AutomationBench README (zapier/AutomationBench on GitHub): https://github.com/zapier/AutomationBench (fetched 2026-10-18)
- Introducing Claude Opus 5.5 (Anthropic, September 22, 2026): https://www.anthropic.com/news/claude-opus-5-5 (fetched 2026-10-18)

Source: https://tessary.ai/answers/automationbench/what-does-an-automationbench-score-mean
More on Automationbench: https://tessary.ai/answers/automationbench
From Tessary, agent reliability for AI agents in production: https://tessary.ai