# LLM as judge

LLM-as-judge is a prompt that grades an agent's output by reasoning over it against one or more
specific intents, the things the agent was supposed to do. The judge reads the output, works through
whether each intent was met, and returns a verdict. It exists because many of the qualities that
matter, like tone, helpfulness, or whether an answer actually addresses the question, need judgment
to assess, and human judgment is too slow and expensive to apply to every output.

A judge's verdicts come from the model its prompt runs on, so they carry that model's properties.
They're stochastic: run the judge twice on the same input and it can disagree with itself. And they
depend on the model version: change the model and the verdicts change meaning, even with the prompt
held fixed.

LLM-as-judge fits checks that need judgment: whether an answer is grounded in the retrieved context,
whether the agent followed a policy, whether the tone fits. Because it costs the most per verdict of
any grader, it usually runs on a subset rather than every output: against a fixed dataset in evals
before a change ships, and in production on traces that were sampled or flagged by cheaper checks.

## Questions answered under this concept

- [Did my agent get worse, or did my judge?](https://tessary.ai/answers/llm-as-judge/agent-worse-or-judge-worse)
- [Does an LLM judge know when it's unsure?](https://tessary.ai/answers/llm-as-judge/does-llm-judge-know-when-unsure)
- [Does pinning the judge model version stop it from drifting?](https://tessary.ai/answers/llm-as-judge/does-pinning-judge-model-stop-drift)
- [Is 90 percent agreement good enough for an LLM judge?](https://tessary.ai/answers/llm-as-judge/is-90-percent-agreement-good-enough)
- [When should I use an LLM judge instead of a deterministic check?](https://tessary.ai/answers/llm-as-judge/when-to-use-llm-judge)
- [Why do judges flip their verdict when you swap the order of two answers?](https://tessary.ai/answers/llm-as-judge/why-judges-flip-verdict-on-answer-order)

---

Source: https://tessary.ai/answers/llm-as-judge
All concepts: https://tessary.ai/answers
From Tessary, agent reliability for AI agents in production: https://tessary.ai