# Does an LLM judge know when it's unsure?

An LLM judge can flag its own uncertainty somewhat reliably: a March 2026 study of seven LLM graders found that asking a model to self-report its confidence produced a lower mean calibration error, 0.166, than sampling the same prompt five times and voting on the result, 0.229, at a fifth of the inference cost.

The catch is in the shape of the scores. Across that study's self-reported and token-probability methods, 86 percent of predictions landed above 0.8 confidence and only 0.62 percent below 0.2, so a judge that's mostly right and one that's mostly wrong hand back confidence scores that look nearly identical. The useful decision threshold isn't near the middle you'd expect.

Self-consistency still earns a place as a second check: run a verdict several times, and a judge that flips on repeat sampling is telling you the item sits near its decision boundary, independent of what any human said about it.

---

Sources:
- March 2026 study of seven LLM graders' confidence calibration: https://arxiv.org/abs/2603.29559 (fetched 2026-08-25)

Source: https://tessary.ai/answers/llm-as-judge/does-llm-judge-know-when-unsure
More on LLM as judge: https://tessary.ai/answers/llm-as-judge
From Tessary, agent reliability for AI agents in production: https://tessary.ai