Can I run AutomationBench on my own agent?

Yes: AutomationBench’s 600 public tasks are free to run yourself through Zapier’s own CLI, installed with uv sync and run as uv run auto-bench --model <model>, but the number you get won’t match the official leaderboard.

The public set splits evenly across six domains, sales, marketing, operations, support, finance, and HR, 100 scored tasks each, plus a separate 200-task “simple” domain that doesn’t count toward the score. The CLI points at a model through an API key or a custom base URL; its README doesn’t describe plugging in your own agent harness, so you’re testing a model against Zapier’s own scaffolding, not necessarily the harness you’d actually ship.

The leaderboard itself grades a separate, held-out private set per domain that Zapier says follows a similar distribution but runs harder, so a local run gives you directional agreement with the leaderboard, not its own number, and there’s no documented way to submit a local result for an official score.