Are most LLM judges reliable?

No. A 2025 benchmark tested 54 LLMs as judges, 43 open-source models from 1 billion to 405 billion parameters plus 11 closed ones, scoring RAG and agentic outputs against human ground truth. Only 27 of the 54, exactly half, reached what the study calls reliable agreement with human judgment: 23 tracked the natural variation between human raters themselves, and 4 agreed even more consistently than humans agree with each other. The other half didn’t clear that bar.

The split didn’t track model size. The study found judge quality came down to how a model was trained to evaluate, not how large it was, so picking the biggest available model isn’t a safe shortcut to a reliable judge. Measuring a specific judge’s own accuracy against your own labeled cases is still the only way to know where the one you picked lands, since a benchmark average says nothing about its performance on your particular grading task.

sources

keep reading

More on this.

Can the content an LLM judge is grading manipulate its verdict?Yes. A 2025 study found text appended to a graded response swayed an LLM judge's verdict over 30% of the time, a documented prompt-injection attack.answer →Can you fix a correlated LLM judge panel?Yes: a 2026 study got 7 to 8 points higher accuracy by modeling which judges share failure patterns, instead of averaging votes as if each judge failed independently.answer →Does an LLM judge score its own model's outputs more favorably?Often, yes, but not from recognizing its own writing: research ties the bias to judges rating familiar-sounding text higher regardless of its source.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y