# Reliabilitybench

ReliabilityBench (arXiv:2601.06112) is a benchmark for tool-using agents that measures
reliability instead of a single-run success rate. It varies three things and reports all three:
how often the same task succeeds on every repeat, how much success drops when the task is
reworded without changing its meaning, and how much it drops when the tools underneath start
failing.

The first axis is consistency. Instead of asking whether the agent can complete a task once, it
runs the same task several times and asks whether it completed every time. An agent that
succeeds on most runs still fails some users, and in production each user only gets one run. So
the score that matters is the one where every attempt has to pass.

The second axis rewords the task. Same request, different phrasing, and the agent should behave
the same. It didn't: at the heavier rewording level, success fell from 96.9% to 88.1%. The third
axis breaks the tools on purpose, with timeouts, rate limits, incomplete responses, and changed
response formats. Rate limits caused the largest drop. In all three, the agent is judged on
whether the job actually got done, not on what it said, so a reworded task still counts as the
same task.

Two caveats before citing it. It's a single-author preprint, not a multi-lab benchmark, and it
covers two models over 1,280 episodes. The measurement design is worth borrowing; the specific
numbers are early.

## Questions answered under this concept

- [How do you measure whether an agent succeeds consistently, not just once?](https://tessary.ai/answers/reliabilitybench/how-do-you-measure-whether-an-agent-succeeds-consistently)
- [How do you test an agent against tool failures?](https://tessary.ai/answers/reliabilitybench/how-do-you-test-an-agent-against-tool-failures)
- [What is fault injection for AI agents?](https://tessary.ai/answers/reliabilitybench/what-is-fault-injection-for-ai-agents)
- [What is ReliabilityBench?](https://tessary.ai/answers/reliabilitybench/what-is-reliabilitybench)

---

Source: https://tessary.ai/answers/reliabilitybench
All concepts: https://tessary.ai/answers
From Tessary, agent reliability for AI agents in production: https://tessary.ai