Does an LLM judge know when it's unsure?

An LLM judge can flag its own uncertainty somewhat reliably: a March 2026 study of seven LLM graders found that asking a model to self-report its confidence produced a lower mean calibration error, 0.166, than sampling the same prompt five times and voting on the result, 0.229, at a fifth of the inference cost.

The catch is in the shape of the scores. Across that study’s self-reported and token-probability methods, 86 percent of predictions landed above 0.8 confidence and only 0.62 percent below 0.2, so a judge that’s mostly right and one that’s mostly wrong hand back confidence scores that look nearly identical. The useful decision threshold isn’t near the middle you’d expect.

Self-consistency still earns a place as a second check: run a verdict several times, and a judge that flips on repeat sampling is telling you the item sits near its decision boundary, independent of what any human said about it.

sources

keep reading

More on this.

Does pinning the judge model version stop it from drifting?Not fully. A pinned version string is a record, not a guarantee, since a hosted provider can serve updated weights under the same stable alias.answer →Does running more LLM judges on the same trace reduce error?Not by much: a 2026 study found nine LLM judges carried only about two judges' worth of independent signal, since they fail on the same items.answer →How does an LLM judge compare to human evaluation?About as often as two human graders agree with each other: 85 percent versus 81 percent on MT-Bench, the benchmark that established LLM-as-judge as a working method.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y