# How do you measure whether an agent succeeds consistently, not just once?

Run the task multiple times and score whether every run passes, not whether any one of them does. ReliabilityBench calls this metric pass^k, distinct from the pass@1 rate a single attempt gives you: in its Travel and E-commerce domains, a ReAct agent that succeeds on any given try 87.5% of the time succeeds on both of two straight tries only 75% of the time. The paper's own conclusion is that pass@1 overestimates an agent's real reliability by 20 to 40%, because [a nondeterministic agent](/answers/agent-reliability/why-does-the-same-prompt-give-different-answers) can pass a task once through luck it won't repeat.

The gap isn't fixed. The benchmark's Scheduling domain hit 100% on both pass@1 and pass2, so some tasks really are that stable; Support sat closer to the Travel and E-commerce pattern, 91.7% down to 83.3%. Read the method as the useful part, scoring consistency across repeats rather than a single pass, and expect the size of the gap to depend on the task rather than the benchmark. It's a single-author preprint over two models and 1,280 episodes, early evidence for the approach rather than a settled number.

---

Sources:
- ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions (arXiv:2601.06112): https://arxiv.org/abs/2601.06112 (fetched 2026-09-12)

Source: https://tessary.ai/answers/reliabilitybench/how-do-you-measure-whether-an-agent-succeeds-consistently
More on Reliabilitybench: https://tessary.ai/answers/reliabilitybench
From Tessary, agent reliability for AI agents in production: https://tessary.ai