What is a grader?

A grader is a check that reads a trace of an agent’s behavior and returns a verdict about one specific thing the agent was supposed to do. Run that same grader across every trace of that behavior and a vague sense of quality becomes a rate: how often it passes now, whether that held after your last change, whether it’s worse on a new model version. A rate compares across agent versions and over time; a general impression of quality does not.

Graders come in three forms, and cost rises with how much judgment the check requires. A deterministic check is code: did the tool call error, does the output parse, is a required field present. It’s cheap enough to run on every trace. A trained classifier reads the text and scores one property at low cost, without reasoning about it. An LLM judge reasons about meaning, the only way to catch a reply that contradicts what the user said three turns earlier, and it costs the most per verdict.

Pick the cheapest form that can still tell the truth about the specific behavior you’re checking.