Reliabilitybench

ReliabilityBench (arXiv:2601.06112) is a benchmark for tool-using agents that measures reliability instead of a single-run success rate. It varies three things and reports all three: how often the same task succeeds on every repeat, how much success drops when the task is reworded without changing its meaning, and how much it drops when the tools underneath start failing.

The first axis is consistency. Instead of asking whether the agent can complete a task once, it runs the same task several times and asks whether it completed every time. An agent that succeeds on most runs still fails some users, and in production each user only gets one run. So the score that matters is the one where every attempt has to pass.

The second axis rewords the task. Same request, different phrasing, and the agent should behave the same. It didn't: at the heavier rewording level, success fell from 96.9% to 88.1%. The third axis breaks the tools on purpose, with timeouts, rate limits, incomplete responses, and changed response formats. Rate limits caused the largest drop. In all three, the agent is judged on whether the job actually got done, not on what it said, so a reworded task still counts as the same task.

Two caveats before citing it. It's a single-author preprint, not a multi-lab benchmark, and it covers two models over 1,280 episodes. The measurement design is worth borrowing; the specific numbers are early.