Do agent benchmarks score good judgment calls, or just whether the task got done?

Mostly just the outcome. A 2026 audit scored 15 widely used agent benchmarks across six coverage axes: Control, Trajectory, Tool execution, Security, Memory/state, and Efficiency. Control is the axis that bundles together six kinds of mid-task judgment call: acting, asking, refusing, stopping, confirming before an irreversible step, and recovering from a mistake. Only two benchmarks, tau-bench and ToolSandbox, score Control directly rather than exercising it as a side effect of a task that happened to need one. Tool execution fared much better, with 9 of the 15 scoring it directly, and efficiency fared worst: not one of the 15 scored it at all. A benchmark that only grades the end state can’t tell a clean success from one where the agent got lucky after a bad call along the way, which is the same reason a completion grader checks the state a session actually left behind instead of trusting what the transcript says happened. The paper calls its own numbers audit judgments from a single protocol, not an independently replicated study, so read them as a map of what’s missing rather than a precise score.

See also: Why do AI agents lie on Vending-Bench?