Does a high AutomationBench score mean an agent will work in production?

A high score on Zapier’s AutomationBench shows a model can finish hard cross-app workflows in a simulation, which isn’t the same as working in your production stack. Zapier’s whitepaper (arXiv:2604.18934) says app behavior “does not represent exact production behavior”: each app is Pydantic models behind real API shapes, and no authentication is mimicked.

The run conditions are narrower than production too. Each task is a single instruction and then “non-interactive until completion,” so the agent can’t ask a clarifying question, and the whitepaper says scores are for a single run. Its run-to-run variance, “typically within 1%,” is a figure for the overall pass rate; the paper doesn’t report whether the same task passes on every run, and that per-task consistency is what a single-attempt score overstates.

The score also doesn’t tell you which of your workflows a model handles. On the whitepaper’s April 2026 leaderboard, Anthropic’s Claude Opus 4.7 scored 9.9% and Google’s Gemini 3.1 Pro 9.6%, yet their passing sets had a Jaccard similarity of 0.17, and 71% of the tasks Opus solved, Gemini didn’t. The tasks themselves are synthetic, and the authors’ own limitations section names a “risk of lack of realism and impossibility.”