Model Version Regression Evals: A Pre-Upgrade Checklist

A model version bump ships in a config value. No PR touches the prompt, the tool schema, or the application code, and the agent starts declining refund requests it used to handle. The first signal is a customer email. Model version regression evals exist so the first signal is a grader instead: run your existing graders against the outgoing and incoming model on the same production traces, and compare the verdicts before the swap reaches users.

The reason a model swap is harder to catch than a prompt edit is visibility. A prompt edit shows up in a pull request diff; someone can read it and reason about what changed. A model version bump, say from one Claude or GPT snapshot to the next, changes what happens inside a prompt string that never moved. Nothing in your codebase tells you the refusal threshold shifted, or that the new model now follows an instruction more literally than the one it replaced.

Why a Model Version Bump Breaks Things a Prompt Diff Won’t Show

A model version regression has no line to review. The prompt is unchanged, the tool schema is unchanged, the surrounding code is unchanged. The only thing different is which model is interpreting all of it, and that difference lives in a config value or an API parameter, not in a diff anyone reads before merging.

This is why teams that gate releases on prompt diffs alone get surprised by model upgrades. The upgrade often ships through a config change or a provider default, not a pull request that touches application code. If your eval suite only reruns on prompt changes, a model version bump slips past it by design.

What to Cover in Model Version Regression Evals Before You Upgrade

The checklist is shorter than it sounds, because most of what matters is already covered by an existing regression eval suite, if you point it at the right target. Anthropic’s own engineering guidance on agent evals notes that once a baseline eval suite exists, latency, token usage, cost per task, and error rates come with it for free, tracked against a static bank of tasks. That baseline is what a model version comparison runs against.

Before flipping the version in production, run the full suite against the new model on the same call sites you already track:

• Tool-calling and function selection, since a new model can pick a different tool for the same request
• Refusal and safety thresholds, since a stricter or looser model changes what gets declined
• Output format and schema adherence, since downstream parsers expect a specific shape
• Instruction-following strictness, since a model that follows instructions more literally can break logic that relied on the old model’s looser interpretation

Comparing Verdict Distributions Between the Old and New Model

A single aggregate pass rate hides the failure that matters. An aggregate score of 0.90 can look fine while one call site, like an escalation-routing check, drops 20 points underneath it.

The fix is comparing verdict distributions, not scores. Run the same graders against the outgoing and incoming model version on identical production traces, then look at the pass and fail counts per call site rather than one number. Regression detection built this way treats the model version as one more variable a grader can be pointed at, alongside prompt version and code changes, so the comparison shows exactly which call site moved and by how much. The same comparison works across providers, not only across versions of one model; multi-provider agent evals covers running one grader set against models from different providers.

When the Verdict Distribution Says It’s Safe to Ship

Anthropic frames regression evals plainly: they should ask whether the agent still handles what it used to, and pass close to 100% of the time. A model version upgrade that drops that number on any call site is not ready, no matter how the new model scores on public benchmarks.

Ship when the new version holds pass rate parity or improves on every call site you track, with no new spikes in fail or borderline verdicts on the highest-risk ones, tool selection and any action that touches a refund, cancellation, or account change. If one call site regresses while the rest improve, that is not a wash. It is the one place where the old model’s behavior was doing something the new model isn’t, and benchmarks will not tell you which one it is.

When You Didn’t Choose the Upgrade

Everything above assumes an upgrade moment: a day you decide to move and a window to test in. Often there isn’t one. Pinned snapshots reach end of life and force a migration on the provider’s schedule. An alias like -latest moves to a new snapshot with nothing changing in your config. Providers revise a model behind a stable version string. In each of these, the model changed underneath you, and no pre-upgrade suite ran because nothing on your side marked an upgrade.

For that case, the check has to live in production. Graders score live traces continuously, and the signal is the trend across their verdicts rather than any one run: even a grader that gets individual calls wrong does so at a roughly constant rate, so a fail rate that sits at 2% for a month and climbs to 9% over a week marks a real shift.

Detection still leaves the on-call engineer guessing. A dashboard showing the pass rate fell on July 3 leaves the prompt edit, the dependency bump, and the model bump that all landed that week as equal suspects. That is where score-tracking tools like Langfuse and Braintrust stop: they hold the traces and show that a score moved. Tessary’s attribution is the step after, tracing the failing quality dimension back through the agent’s chain until it can name the cause, in this case the model version, per call site, with no diff to read because the causing change never touched your repo.

Point your graders at both model versions before an upgrade you chose, and when the model moves underneath you, find out that it was the model and not your prompt. No credit card to connect a trace source. Find what broke your agent