# Can tau-bench grade a failing run as a pass?

Only through a specific bug, and not in tau-bench's own official results. An October 2026 audit found that tau-bench's reward function resets the task database to build its own ground-truth hash and never restores it afterward. tau-bench's runner reads that function's return value once and moves on, so its published scores are unaffected. The bug turns into a wrong label only for a separate harness that re-scores a recorded episode after the session ends, because that second read finds the reference solution's leftover state instead of what the agent actually did. The audit built exactly that kind of harness to check tau-bench's reliability, and across 4,320 archived episodes it had already scored, 4,194 were recorded as a database pass; replaying the same tool calls against a clean database found only 2,883, dropping measured success from 94.3% to 65.5%. A different kind of integrity check shows up in [ReliabilityBench's fault injection](/answers/reliabilitybench/what-is-fault-injection-for-ai-agents): it breaks the tools on purpose to see how an agent copes, rather than auditing whether the benchmark's own grading code tells the truth. A benchmark's own grading code can carry a bug its leaderboard never surfaces, because nothing routinely re-tests the oracle itself.

---

Sources:
- When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark (arXiv:2610.16067): https://arxiv.org/abs/2610.16067 (fetched 2026-10-15)

Source: https://tessary.ai/answers/tau-bench/can-tau-bench-grade-a-failing-run-as-a-pass
More on Tau bench: https://tessary.ai/answers/tau-bench
From Tessary, agent reliability for AI agents in production: https://tessary.ai