# Do agent benchmarks score good judgment calls, or just whether the task got done?

Mostly just the outcome. A 2026 audit scored 15 widely used agent benchmarks across six coverage axes: Control, Trajectory, Tool execution, Security, Memory/state, and Efficiency. Control is the axis that bundles together six kinds of mid-task judgment call: acting, asking, refusing, stopping, confirming before an irreversible step, and recovering from a mistake. Only two benchmarks, tau-bench and ToolSandbox, score Control directly rather than exercising it as a side effect of a task that happened to need one. Tool execution fared much better, with 9 of the 15 scoring it directly, and efficiency fared worst: not one of the 15 scored it at all. A benchmark that only grades the end state can't tell a clean success from one where the agent got lucky after a bad call along the way, which is [the same reason a completion grader checks the state a session actually left behind](/answers/failure-modes/how-do-you-catch-an-agent-claiming-a-task-is-done) instead of trusting what the transcript says happened. The paper calls its own numbers audit judgments from a single protocol, not an independently replicated study, so read them as a map of what's missing rather than a precise score.

See also: [Why do AI agents lie on Vending-Bench?](/answers/vending-bench/why-do-ai-agents-lie-on-vending-bench)

---

Sources:
- "AgentAtlas: Beyond Outcome Leaderboards for LLM Agents" (arXiv:2605.20530): https://arxiv.org/abs/2605.20530 (fetched 2026-10-14)

Source: https://tessary.ai/answers/agent-benchmarks/do-agent-benchmarks-score-judgment-calls-or-just-the-outcome
More on Agent benchmarks: https://tessary.ai/answers/agent-benchmarks
From Tessary, agent reliability for AI agents in production: https://tessary.ai