Does a regression grader have to be an LLM judge?

No, and most of the time it shouldn’t be. A regression grader exists to catch one specific failure that already happened once, and a deterministic check, code that asserts the exact condition that broke, a status code, a missing field, a string that shouldn’t appear, is cheaper to run and doesn’t drift the way a judge’s reasoning can drift between model versions.

An LLM judge earns its cost when the failure isn’t a fact you can assert in code but a matter of meaning: the agent technically answered but missed the point, or contradicted something said earlier in the conversation. That kind of failure needs a model reading the trace and reasoning about it, because no regex or field check can tell you the answer was off topic.

Write the deterministic check first. Reach for a judge only when the check keeps passing on traces you know are still broken.

keep reading

More on this.

What stops a description from producing a grader that measures the wrong thing?Nothing by itself. A description only encodes what its author imagined the failure looks like, and needs grounding in a real trace where that failure actually happened before it ships.answer →How do you know when a grader no longer matches what the agent does?Not from the pass rate. Check whether the prompt, tool contract, or output schema the grader was written against has changed since, because it keeps judging the old version either way.answer →What does an eval score actually claim about my agent?It's a claim about the eval set the graders ran on, not the agent as a whole; it says nothing about a failure mode no grader in the set checks for.answer →Everything on gradersThe definition, and every question answered under graders.concept →

Send us the traces you already emit.

book a demo