Agent Regressions from Non-Prompt Code Changes

Anthropic’s April 2026 engineering postmortem documented two infrastructure changes that degraded Claude Code for weeks. Neither touched a prompt. A session caching bug cleared reasoning context every turn instead of once. A configuration change dropped the default reasoning effort from high to medium. Neither change appeared as a measurable failure in the existing evals, and users experienced both as the model getting worse.

Part of why both stayed invisible is that nothing in the traces captured session state or the reasoning-effort setting. A gap like that is informative: when a trace goes dark partway to a cause, the gap itself is the next node to instrument, and the next regression is easier to trace.

This is the class of problem: agent regression from non-prompt code changes.

Why Non-Prompt Code Changes Cause Agent Regressions

A prompt can stay identical across two deploys and the agent can still break. The cause is upstream from the prompt.

A library update alters the schema of what your code passes to the model. A refactored call chain changes what context reaches the prompt assembly step. A dependency update shifts the format of a tool response the model is expected to parse. An infrastructure change modifies how much reasoning context is preserved across turns.

In each case, the model is receiving different inputs or producing outputs that land in different parsing code. The prompt is not the variable. The surrounding code is.

This is why the regression detection problem for agentic products is broader than most eval tooling assumes. Eval pipelines built around prompt-to-output comparison cover one slice of the failure surface. They leave non-prompt code changes uncovered by design.

Why Prompt-Only Evals Miss Non-Prompt Regressions

Standard eval workflows compare outputs against expected results, given the same prompts. That approach catches prompt regressions: if you change the prompt and the behavior degrades, the eval surfaces it.

It does not catch non-prompt regressions.

When a call chain is refactored, the prompt fed to the model might be identical to what ran last week. The surrounding infrastructure is different. A grader that only looks at prompt-to-output consistency does not see the refactor as a variable. It sees the same prompt and an output that looks plausible, without knowing that the context assembly step now produces different content.

Manual trace review has the same gap. An engineer reviewing traces after a library update will compare outputs to expectations. Without knowing the tool response schema changed, they have no baseline for what correct looks like given the new inputs. They are reviewing outputs in isolation from the code change that produced them.

How Graders Catch Agent Regressions from Code Changes

A grader evaluates a specific failure mode at a specific call site. It runs against production traces, which means it operates on real inputs as the system assembled them at call time, not on synthetic prompts constructed outside the production environment.

When a library update changes a tool response schema, the grader sees the new schema in the trace. A grader checking whether an expected field is present in the tool response will fail if the field name changed. A grader checking whether the model output references a retrieved document will fail if the retrieval layer stopped returning it. These graders catch the regression not by comparing to the old prompt, but by checking what the model did with what it received.

This is why trace-grounded graders detect non-prompt regressions where prompt comparisons do not: they evaluate the full call, including the surrounding code’s contribution to the inputs.

Braintrust’s CI/CD evaluation guide recommends the same cadence: run graders after every deploy, not only when prompts change. That cadence gets you detection. Running graders per deploy tells you a score moved after the deploy; it does not tell you which change moved it. A real release bundles a prompt tweak, a dependency bump, and a refactor into one deploy, so a flipped grader still leaves every change in that release as a suspect. The next section is about closing that gap.

Running Graders After Every Deploy, Not Just Prompt Changes

The practical implication is that graders need to run on production traces after every deploy, regardless of whether a prompt changed.

A working setup connects your trace source, runs the eval suite against spans from the latest deploy, and compares pass and fail counts to the prior deploy baseline. Read the movement across verdicts rather than any single flip: a grader that captures intent imperfectly still misfires at a steady rate, so a fail rate that holds at 4% for weeks and climbs to 12% after a deploy signals a real regression to investigate before more traffic runs through it.

Investigation is where the manual version stalls, because the deploy that moved the rate rarely contains one change. This is the attribution step: when a grader starts failing, it names the quality dimension that broke, and Tessary traces that dimension back through the agent’s chain, through the tool response, the context assembly step, the dependency that changed, to the specific change that caused it. A regression to investigate becomes a regression with a named cause. The same engine can run against a change before it merges; pre-deploy failure prediction covers that case.

One maintenance problem remains: keeping graders pointed at the right call sites as the codebase changes. When the call chain is refactored, graders that referenced the old structure become stale, and the suite keeps reporting pass on a code path it is no longer correctly checking. Tessary drafts graders from the agent’s observed behavior, you correct them rather than write them, and it re-drafts as the code changes, so coverage tracks the current call structure rather than the one that existed at launch.

The maintenance story matters, but it is not the point of the tool. The point is the question this article started with: when your agent gets worse after a deploy and no prompt changed, Tessary tells you which library update, schema change, or refactor caused it, fast enough to fix it before it does damage.

Connect your traces and find out which change in your last deploy moved your scores. No credit card needed to start. Find what broke your agent