What is LLM judge drift?

LLM judge drift is a change in how an LLM-as-judge grader scores the same input, caused by the model behind the judge's API changing rather than anything about the system it's grading. A silent provider version bump or an updated scoring prompt can shift a judge's verdicts on identical traces, so a falling eval score can reflect a stricter or looser judge instead of a worse agent. Because both look the same from the score alone, telling them apart requires checking the judge's own consistency, not just watching the trend it produces.

How is judge drift different from a normal eval score drop?

A normal eval score drop assumes the measuring instrument stayed fixed while the thing being measured got worse. Judge drift breaks that assumption: the instrument itself changed. The two produce an identical signal, a lower score over the same window, so a team that only watches the trend has no way to know from the dashboard alone whether to investigate the agent or the grader. The distinction only becomes visible once you re-score something you know the answer to and see whether the judge still agrees with itself.

What causes an LLM judge to drift silently?

Two causes show up in practice: a provider-side model swap that leaves the judge's API version string untouched, or a scoring-prompt edit the team running the judge never registers as a new judge version. Both change how the judge reads the exact same input without any corresponding change in the system being graded, and neither leaves a mark on the repo being deployed, since the judge typically runs outside it.

How do you tell if your agent regressed or your judge changed?

Keep a fixed set of production traces with a known, human-agreed verdict, and re-run the current judge against that set on the same schedule you grade live traffic. If the judge's agreement with those known answers holds steady while the live score drops, the agent changed. If the judge starts disagreeing with answers it used to get right, the judge changed. A June 2026 paper formalizes this as an anchor set: a fixed reference the judge re-scores at a steady interleave, checked against a separate statistical test from the one watching live traffic, so the two questions don't contaminate each other.

Does a stricter grading prompt count as judge drift?

Yes. Judge drift covers any change in how the judge scores, regardless of whether the provider updated the underlying model or a team member edited the scoring prompt. A June 2026 paper tested both cases: a silent provider version bump and a deliberately stricter scoring prompt. Both were correctly identified as judge-side changes rather than system regressions, though the stricter-prompt case was harder to isolate cleanly than the silent version bump.

LLM Judge Drift: How to Tell What Actually Broke

By Akhil Varma · July 25, 2026

Short answer

LLM judge drift happens when the model behind a hosted grader changes behind its own API, so a falling eval score can mean the agent regressed or the judge did. A June 2026 paper resolves the ambiguity with a fixed, human-labeled anchor set the current judge re-scores on a steady interval, isolating which side actually moved.

A June 2026 paper testing production eval pipelines found that the industry-default way of watching a score for drift, a rolling z-test against a threshold, misfires constantly on streams where nothing had actually changed. That’s a detection problem, and the paper solves it. The harder problem underneath it is that even a correct alarm cannot say what moved: the paper’s real subject is LLM judge drift, where the model behind a hosted grader changes behind its own API and every eval trend it produces becomes ambiguous between a worse agent and a worse judge.

Why LLM Judge Drift Looks Identical to a Real Regression

A score dropping over a production window usually gets read one way: the agent got worse. That reading assumes the thing doing the scoring stayed fixed while the thing being scored didn’t. An LLM-as-judge grader is itself a model sitting behind an API, and a provider can update that model, or a team can edit its scoring prompt, without either change touching the system under test. Both a genuine regression and a genuine judge change produce the same artifact: a lower score, over the same window, with nothing in a deploy log to explain it.

The paper (arXiv:2606.15474) treats this as the actual failure mode worth solving, not a footnote to detection. Its authors frame it plainly: a cheap judge scores every interaction and pages a team when the score drifts down, but the judge behind that scoring is a model too, and nothing about the setup can tell you which side of the grading relationship actually changed.

How an Anchor Set Separates a Worse Agent From a Worse Judge

The paper’s fix is a second, smaller measurement running alongside the main one. A fixed set of production traces gets a human-agreed verdict once, and the current judge re-scores that same fixed set on a steady interleave, separate from the live traffic it’s grading. A second statistical process watches only the gap between the judge and those known answers, kept independent from the process watching the live score, so a change in one can’t be mistaken for a change in the other. When the two disagree about which side moved, a guard-window rule returns one of three verdicts: nothing changed, the system changed, or the judge changed.

On two real judge changes, the method caught a silent version bump as judge drift in 60 of 60 runs with zero cases misattributed to the system. A deliberately stricter scoring prompt, a harder case because the change was designed to look plausible rather than accidental, was still correctly attributed to the judge in 110 of 120 runs at the paper’s tested guard width.

Compare that to the 75% false-alarm rate from the plain threshold test with no anchor set at all: without a way to check the judge against something fixed, a team running the industry-default method is wrong about drift three times out of four, before even getting to the harder question of which side caused it.

What This Means for Reading Your Own Eval Trend

Most teams running an LLM-as-judge grader in production have exactly the live-traffic score and nothing else: no fixed reference the judge re-scores, no separate check on whether the judge itself still agrees with a known answer. A dropping score gets treated as agent news by default, because there’s no mechanism in place that could tell you otherwise. The fix isn’t complicated in principle, keep a labeled set the judge periodically re-grades, but it’s a second pipeline most teams don’t build until a drop turns out to have been the judge all along.

Tessary’s job is the same one this paper is solving for a narrower case: when an agent gets worse in production, name the change responsible before the drop turns into a support queue. That search runs into the same ambiguity the paper describes.

A grader trend breaking could mean the agent regressed, or it could mean the judge scoring it drifted, and a provider updating the model behind a grader belongs to the same no-diff category as a provider updating the model behind the agent itself or an upstream service returning different data: none of it touched a repo or a deploy pipeline. Ruling the judge in or out has to happen before the agent takes the blame, not after.

That’s also why a single grader verdict isn’t the thing worth watching. Tessary reads the trend across a grader’s verdicts over time, not any one score: an imperfect judge misfires at a stable rate, so a judge whose normal fail rate sits near 6% and triples inside a week is signaling a real change, not the background noise its own baseline already accounts for.

The graders behind that search start as a first pass Tessary drafts from what the agent has actually done in production, then get sharpened by whoever owns the intent, closer to editing a working draft than assembling an eval suite line by line.

Point Tessary at a judge you already run and see whether last week’s drop traces back to your agent or to the model grading it, no credit card required to look. For calibrating a judge against human labels in the first place, LLM judge calibration covers the row-by-row check; for what changes once a grader is live, see how evals evolve past the first version.

Find what broke your agent

Frequently asked questions

What is LLM judge drift?: LLM judge drift is a change in how an LLM-as-judge grader scores the same input, caused by the model behind the judge's API changing rather than anything about the system it's grading. A silent provider version bump or an updated scoring prompt can shift a judge's verdicts on identical traces, so a falling eval score can reflect a stricter or looser judge instead of a worse agent. Because both look the same from the score alone, telling them apart requires checking the judge's own consistency, not just watching the trend it produces.
How is judge drift different from a normal eval score drop?: A normal eval score drop assumes the measuring instrument stayed fixed while the thing being measured got worse. Judge drift breaks that assumption: the instrument itself changed. The two produce an identical signal, a lower score over the same window, so a team that only watches the trend has no way to know from the dashboard alone whether to investigate the agent or the grader. The distinction only becomes visible once you re-score something you know the answer to and see whether the judge still agrees with itself.
What causes an LLM judge to drift silently?: Two causes show up in practice: a provider-side model swap that leaves the judge's API version string untouched, or a scoring-prompt edit the team running the judge never registers as a new judge version. Both change how the judge reads the exact same input without any corresponding change in the system being graded, and neither leaves a mark on the repo being deployed, since the judge typically runs outside it.
How do you tell if your agent regressed or your judge changed?: Keep a fixed set of production traces with a known, human-agreed verdict, and re-run the current judge against that set on the same schedule you grade live traffic. If the judge's agreement with those known answers holds steady while the live score drops, the agent changed. If the judge starts disagreeing with answers it used to get right, the judge changed. A June 2026 paper formalizes this as an anchor set: a fixed reference the judge re-scores at a steady interleave, checked against a separate statistical test from the one watching live traffic, so the two questions don't contaminate each other.
Does a stricter grading prompt count as judge drift?: Yes. Judge drift covers any change in how the judge scores, regardless of whether the provider updated the underlying model or a team member edited the scoring prompt. A June 2026 paper tested both cases: a silent provider version bump and a deliberately stricter scoring prompt. Both were correctly identified as judge-side changes rather than system regressions, though the stricter-prompt case was harder to isolate cleanly than the silent version bump.

Written by

Akhil Varma · Founder, Tessary

Akhil builds Tessary — AI personas that run real-browser usability tests on B2B SaaS products. Previously shipped product at multiple early-stage startups; writes about usability testing, AI personas, and the economics of B2B research.