Does a regression grader have to be an LLM judge?

No, and most of the time it shouldn’t be. A regression grader exists to catch one specific failure that already happened once, and a deterministic check, code that asserts the exact condition that broke, a status code, a missing field, a string that shouldn’t appear, is cheaper to run and doesn’t drift the way a judge’s reasoning can drift between model versions.

An LLM judge earns its cost when the failure isn’t a fact you can assert in code but a matter of meaning: the agent technically answered but missed the point, or contradicted something said earlier in the conversation. That kind of failure needs a model reading the trace and reasoning about it, because no regex or field check can tell you the answer was off topic.

Write the deterministic check first. Reach for a judge only when the check keeps passing on traces you know are still broken.