Why does an agent that passes a benchmark still fail in production?

Because the number a benchmark reports is usually a best case: one clean attempt, or a few, against tools that behave. ReliabilityBench’s own numbers show both gaps. An agent that succeeds on any single try 87.5% of the time only clears two straight tries 75% of the time, so a pass rate from one attempt overestimates real reliability by 20 to 40%: it measures whether the agent can succeed, not whether it succeeds every time a real user asks. Separately, the same kind of agent’s score drops to 93.75% once its tools start rate-limiting, the one injected fault that cuts deeper than a clean run, because an agent that handles backoff poorly abandons the task instead of retrying.

Neither gap shows up in a single run against cooperative tools, which is what most benchmark numbers measure. A benchmark score also doesn’t carry over to your own scaffold even before repetition or tool failure enter the picture, so the two problems compound rather than replace each other.

See also: Does a high AutomationBench score mean an agent will work in production?