When should I use an LLM judge instead of a deterministic check?

Reach for an LLM judge when grading needs judgment: whether an answer actually addresses what was asked, whether a tool call was justified given the context, whether a multi-step plan held together. A rule or a classifier doesn’t answer those reliably, because the criterion is a reading of intent against context, not a fixed pattern.

Skip it when a deterministic check exists instead. A schema violation, a missing field, or a bad status code is faster and cheaper for code to catch, and a judge call only adds cost and stochastic noise on top of that. A judge is also the most expensive grader per verdict, so most teams don’t run it on every output: a fixed dataset before a change ships, and a sampled or flagged slice of production traffic rather than the full stream.