Why do vision agents need a grader that checks perception and action separately?

Because an agent can describe an image correctly and act on the wrong part of it, or describe it wrong and still act in a way that reads as reasonable, and a single verdict can’t tell you which happened. Agent-X, a 2025 benchmark of vision-centric agent tasks across web browsing, driving, and surveillance, requires every step of a task, perception and action together, to be right, and found that the GPT, Gemini, and Qwen model families all finish under half of them. Scoring only the final action produces that one number and throws away which half broke it.

A perception grader checks the agent’s claim about the image against a reference for what the image actually showed. An action grader checks whether what the agent did follows from what it claimed to see. A misread screenshot that still reads as a confident, fluent reply is a silent failure exactly because nothing in the response format signals which of the two graders would have caught it.