Does an LLM judge know when it's unsure?

An LLM judge can flag its own uncertainty somewhat reliably: a March 2026 study of seven LLM graders found that asking a model to self-report its confidence produced a lower mean calibration error, 0.166, than sampling the same prompt five times and voting on the result, 0.229, at a fifth of the inference cost.

The catch is in the shape of the scores. Across that study’s self-reported and token-probability methods, 86 percent of predictions landed above 0.8 confidence and only 0.62 percent below 0.2, so a judge that’s mostly right and one that’s mostly wrong hand back confidence scores that look nearly identical. The useful decision threshold isn’t near the middle you’d expect.

Self-consistency still earns a place as a second check: run a verdict several times, and a judge that flips on repeat sampling is telling you the item sits near its decision boundary, independent of what any human said about it.

sources

keep reading

More on this.

Did my agent get worse, or did my judge?You can't tell from a falling score alone; run a fixed, human-labelled anchor set through the current judge on a steady interleave and watch it separately.answer →Does pinning the judge model version stop it from drifting?Not fully. A pinned version string is a record, not a guarantee, since a hosted provider can serve updated weights under the same stable alias.answer →Is 90 percent agreement good enough for an LLM judge?Not on its own. Raw agreement doesn't correct for chance, so on a skewed label set 90 percent can sit on a kappa near zero.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Send us the traces you already emit.

book a demo