What is ReliabilityBench?

ReliabilityBench is a benchmark that scores tool-using agents on reliability rather than a single run’s success rate. It varies three things and reports all three: how often the same task succeeds on every repeat, how much success drops when the task is reworded without changing its meaning, and how much it drops when the underlying tools start failing with timeouts, rate limits, or malformed responses. Rewording alone pulled one tested setup from 96.9% success to 88.1%, and injected rate-limit failures caused the largest drop of any fault tried.

The point of the second and third axes is that a real user’s phrasing and a real API’s uptime aren’t the benchmark’s to control, so an agent that only works when both cooperate isn’t reliable by this measure even if it passes on the first try. Read the numbers as an early, single-author result rather than an industry baseline: it covers two models and two agent architectures over 1,280 episodes. The measurement approach, scoring consistency instead of one attempt, matters more right now than today’s specific scores.