How much does pass@1 overstate how reliable an agent really is?

By a lot: a 2026 benchmark of stateful business workflows found Claude Opus 5’s success rate fall from 66.5% on a single attempt (pass@1) to 47.53% once the same task had to succeed on all twenty repeats (pass^20), and Kimi-K3 fall from 57.37% to 17.60%. Thinkingbox-bench scores 507 policy-conditioned workflows, retail, hospitality, auto insurance, neobank IT, and consulting IT or HR support, by checking the terminal state a tool session is left in rather than any single response, with task-specific checks that reject a wrong, missing, or extra effect. Pass@1 asks whether one attempt succeeded; pass^k, the same strict metric tau-bench uses, asks whether every one of k repeated attempts did, and the gap between the two is the share of passes that depended on luck rather than a correct, repeatable procedure. A second finding compounds it: many failed trials end cleanly, after the agent took a real, valid state-changing action, so a check that only looks at whether a step looked reasonable can’t tell a genuine completion from one of these.

See also: What does a Vending-Bench score mean?