# Why does an agent that passes a benchmark still fail in production?

Because the number a benchmark reports is usually a best case: one clean attempt, or a few, against tools that behave. ReliabilityBench's own numbers show both gaps. An agent that succeeds on any single try 87.5% of the time only clears two straight tries 75% of the time, so a pass rate from one attempt overestimates real reliability by 20 to 40%: it measures whether the agent can succeed, not whether it succeeds every time a real user asks. Separately, the same kind of agent's score drops to 93.75% once its tools start rate-limiting, the one injected fault that cuts deeper than a clean run, because an agent that handles backoff poorly abandons the task instead of retrying.

Neither gap shows up in a single run against cooperative tools, which is what most benchmark numbers measure. [A benchmark score also doesn't carry over to your own scaffold](/answers/agent-benchmarks/what-is-an-agent-benchmark) even before repetition or tool failure enter the picture, so the two problems compound rather than replace each other.

See also: [Does a high AutomationBench score mean an agent will work in production?](/answers/automationbench/does-a-high-automationbench-score-mean-an-agent-will-work-in-production)

---

Sources:
- ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions (arXiv:2601.06112): https://arxiv.org/abs/2601.06112 (fetched 2026-10-04)

Source: https://tessary.ai/answers/reliabilitybench/why-does-an-agent-that-passes-a-benchmark-still-fail-in-production
More on Reliabilitybench: https://tessary.ai/answers/reliabilitybench
From Tessary, agent reliability for AI agents in production: https://tessary.ai