# Does a high AutomationBench score mean an agent will work in production?

A high score on Zapier's AutomationBench shows a model can finish hard cross-app workflows in a simulation, which isn't the same as working in your production stack. Zapier's whitepaper (arXiv:2604.18934) says app behavior "does not represent exact production behavior": each app is Pydantic models behind real API shapes, and no authentication is mimicked.

The run conditions are narrower than production too. Each task is a single instruction and then "non-interactive until completion," so the agent can't ask a clarifying question, and the whitepaper says scores are for a single run. Its run-to-run variance, "typically within 1%," is a figure for the overall pass rate; the paper doesn't report whether the same task passes on every run, and that per-task consistency is [what a single-attempt score overstates](/answers/agent-benchmarks/how-much-does-pass-1-overstate-agent-reliability).

The score also doesn't tell you which of your workflows a model handles. On the whitepaper's April 2026 leaderboard, Anthropic's Claude Opus 4.7 scored 9.9% and Google's Gemini 3.1 Pro 9.6%, yet their passing sets had a Jaccard similarity of 0.17, and 71% of the tasks Opus solved, Gemini didn't. The tasks themselves are synthetic, and the authors' own limitations section names a "risk of lack of realism and impossibility."

---

Sources:
- Daniel Shepard and Robin Salimans, "AutomationBench" (arXiv:2604.18934, April 21, 2026): https://arxiv.org/html/2604.18934v1 (fetched 2026-10-18)
- AutomationBench leaderboard and methodology, version 1.0.6 (Zapier): https://zapier.com/benchmarks (fetched 2026-10-18)

Source: https://tessary.ai/answers/automationbench/does-a-high-automationbench-score-mean-an-agent-will-work-in-production
More on Automationbench: https://tessary.ai/answers/automationbench
From Tessary, agent reliability for AI agents in production: https://tessary.ai