# How much does pass@1 overstate how reliable an agent really is?

By a lot: a 2026 benchmark of stateful business workflows found Claude Opus 5's success rate fall from 66.5% on a single attempt (pass@1) to 47.53% once the same task had to succeed on all twenty repeats (pass^20), and Kimi-K3 fall from 57.37% to 17.60%. Thinkingbox-bench scores 507 policy-conditioned workflows, retail, hospitality, auto insurance, neobank IT, and consulting IT or HR support, by checking the terminal state a tool session is left in rather than any single response, with task-specific checks that reject a wrong, missing, or extra effect. Pass@1 asks whether one attempt succeeded; pass^k, [the same strict metric tau-bench uses](/answers/tau-bench/how-is-tau-bench-scored), asks whether every one of k repeated attempts did, and the gap between the two is the share of passes that depended on luck rather than a correct, repeatable procedure. A second finding compounds it: many failed trials end cleanly, after the agent took a real, valid state-changing action, so a check that only looks at whether a step looked reasonable can't tell a genuine completion from one of these.

See also: [What does a Vending-Bench score mean?](/answers/vending-bench/what-does-a-vending-bench-score-mean)

---

Sources:
- One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741): https://arxiv.org/abs/2608.19741 (fetched 2026-10-15)

Source: https://tessary.ai/answers/agent-benchmarks/how-much-does-pass-1-overstate-agent-reliability
More on Agent benchmarks: https://tessary.ai/answers/agent-benchmarks
From Tessary, agent reliability for AI agents in production: https://tessary.ai