How do you know when a grader no longer matches what the agent does?

You know by checking whether the prompt, the tool’s contract, or the output schema the grader was written against has changed, not by watching the pass rate. The pass rate alone won’t tell you, because a grader keeps returning verdicts whether or not the thing it was written to check still exists.

A grader that checks “did the agent ask for an order ID before refunding” is silently wrong the day someone removes the order ID requirement, even though every trace after that keeps passing cleanly. The rate looks healthy because the behavior it’s checking for no longer applies to anyone.

Tie the review to the commit, not the calendar: any change that touches the prompt, tool contract, or output schema a grader depends on is the trigger to reopen it, whether or not its rate has moved yet.