LLM as judge

LLM-as-judge is a prompt that grades an agent's output by reasoning over it against one or more specific intents, the things the agent was supposed to do. The judge reads the output, works through whether each intent was met, and returns a verdict. It exists because many of the qualities that matter, like tone, helpfulness, or whether an answer actually addresses the question, need judgment to assess, and human judgment is too slow and expensive to apply to every output.

A judge's verdicts come from the model its prompt runs on, so they carry that model's properties. They're stochastic: run the judge twice on the same input and it can disagree with itself. And they depend on the model version: change the model and the verdicts change meaning, even with the prompt held fixed.

LLM-as-judge fits checks that need judgment: whether an answer is grounded in the retrieved context, whether the agent followed a policy, whether the tone fits. Because it costs the most per verdict of any grader, it usually runs on a subset rather than every output: against a fixed dataset in evals before a change ships, and in production on traces that were sampled or flagged by cheaper checks.