How do you test an agent against tool failures?

Inject the actual failures a production tool can throw at an agent, timeouts, rate limits, and incomplete responses, and measure how much task success drops under each rather than assuming all failures hurt equally. ReliabilityBench’s fault-injection axis found rate limits are the one that actually hurts: isolated rate-limiting cut the pass rate to 93.75%, about 2.5 points below its mixed-fault baseline and the only fault type that dropped below it, while an isolated timeout (98.75%) or partial response (97.50%) barely moved the number.

That ranking is counterintuitive: a rate limit is an explicit 429, as visible an error as a timeout, yet it did more damage. The paper’s explanation is retry behavior: agents handle backoff for a rate-limited API poorly and often abandon the task, while a timeout gets retried and mostly recovers. Tessary’s tool_error classifier would still catch a tool rate-limiting in production, since a 429 is the structural error status it counts; it’s the low-impact partial-response fault, a tool answering with less while reporting success, that slips past a check built on structural evidence. Testing against fixed, injected faults, rather than a tool’s advertised uptime, surfaces the retry gap before it shows up in your own traces.