How do you know when a grader no longer matches what the agent does?

You know by checking whether the prompt, the tool’s contract, or the output schema the grader was written against has changed, not by watching the pass rate. The pass rate alone won’t tell you, because a grader keeps returning verdicts whether or not the thing it was written to check still exists.

A grader that checks “did the agent ask for an order ID before refunding” is silently wrong the day someone removes the order ID requirement, even though every trace after that keeps passing cleanly. The rate looks healthy because the behavior it’s checking for no longer applies to anyone.

Tie the review to the commit, not the calendar: any change that touches the prompt, tool contract, or output schema a grader depends on is the trigger to reopen it, whether or not its rate has moved yet.

keep reading

More on this.

Does a regression grader have to be an LLM judge?No. A deterministic check that asserts the exact condition that broke is cheaper and more stable than an LLM judge, which only earns its cost on failures that need judging meaning.answer →What stops a description from producing a grader that measures the wrong thing?Nothing by itself. A description only encodes what its author imagined the failure looks like, and needs grounding in a real trace where that failure actually happened before it ships.answer →What does an eval score actually claim about my agent?It's a claim about the eval set the graders ran on, not the agent as a whole; it says nothing about a failure mode no grader in the set checks for.answer →Everything on gradersThe definition, and every question answered under graders.concept →

Send us the traces you already emit.

book a demo