What is agent regression test coverage?

A grader written for the exact failure a bug fix addressed, added to the eval suite in the same commit as the patch and run in CI on every future deploy. It differs from general eval coverage, which tests cases the team anticipated. Regression coverage exists specifically because a fix happened, so the eval suite can tell the difference between the state before the fix and the state after it on every subsequent change.

Why do LLM agent bugs come back after being fixed?

An agent bug returns after a fix ships because the fix changed the code or prompt, but no grader was added to check for the specific failure. Without one, a later prompt change, a code refactor, or a model version update that never touches your repo can reintroduce the same behavioral gap in a different form, and nothing is pointed at that failure to flag it.

How do you prevent agent bug recurrence?

Prevent agent bug recurrence by writing the regression test at fix time, not as a follow-up task. Extract the trace that shows the failure, write a grader that checks the failure mode rather than the exact bad output, and add it to the eval suite in the same commit as the fix. Wiring that suite into a CI gate means the next deploy runs the grader automatically, so a reintroduced failure gets caught before it reaches users again.

What causes an agent eval coverage gap?

An agent eval coverage gap happens when the eval suite passes cleanly because it was never built to catch the specific behavior that later broke. Anthropic's own April 2026 postmortem on Claude Code quality issues described running weeks of internal testing with no regressions in their evaluation set, then shipping a change that still caused a measurable quality drop. The suite was accurate for what it covered; the gap was in what it never tested.

When should you write a regression test for an agent bug?

Write the regression test in the same work session as the fix, before closing the ticket. Once the incident's urgency fades, writing the eval competes with new feature work and rarely wins, and the person picking it up later has to reconstruct the failure from a stale description instead of the original trace. Treating the grader as part of the fix, not a follow-up task, is what keeps regression coverage from falling behind.

How is regression coverage different from a broader eval suite?

A broader eval suite covers the inputs a team anticipated when building it. Regression coverage targets a narrower, specific set: the exact behaviors that have already failed in production and been fixed. The two overlap but serve different purposes. A broad suite catches gaps in what you expected; regression coverage guarantees a bug you already paid for once cannot silently reappear on the next change.

All articles

agent regression test coverageLLM agent regression preventionagent eval coverage gapCI pipeline

How to Close the Agent Regression Test Coverage Gap

By Akhil Varma · July 9, 2026

Short answer

Regression coverage for an agent means every bug fix ships with a grader that encodes the specific failure, added in the same commit as the patch and wired into CI. Without it, an eval suite can pass cleanly while the exact behavior a fix addressed regresses again on the next prompt, model, or code change. And when the grader fires, the next question is which change reintroduced the bug; Tessary traces the failing grader back to that change.

Anthropic’s April 2026 postmortem names three product changes that each passed the existing eval suite before shipping, then still caused a measured 3% quality drop, exactly the gap agent regression test coverage exists to catch. On one of them, Anthropic’s postmortem says: “After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change.”

The three changes were a reasoning-effort downgrade, a caching bug that cleared thinking context on every turn instead of once, and a system prompt verbosity limit; users noticed the quality drop before the suite did. This is the agent eval coverage gap: evals that pass while the specific behavior a fix depends on goes unchecked, and it only closes when fixing a bug and writing the test for it happen as one task, not two.

Why an Eval Suite Passing Doesn’t Mean the Fix Is Covered

An eval suite can run clean and still miss the one behavior a specific fix was supposed to protect. On the verbosity change, Anthropic’s own suite reported no regressions across its coding tasks before the change shipped, because none of those cases exercised a 25-word tool-call limit against multi-step coding work.

The suite measured what it was built to measure, and that scope never included the interaction that broke. The same pattern shows up in smaller agents, at smaller stakes than a public postmortem: a team fixes an incorrect tool call, closes the ticket, and the existing eval suite keeps passing because nothing in it exercises the input that triggered the bug. Nothing has to reach the scale of a provider postmortem for the pattern to repeat; a single unlogged ticket is enough.

Regression graders close this gap for failures a team has already seen once. For the ones it has not, detection has to run on production traces, where the trend across grader verdicts flags a drop even when no pre-deploy eval was pointed at it. An imperfect grader misfires at a stable rate, so a moving rate still marks a real change.

Why the Fix and the Test Get Written by Different People, at Different Times

The fix and the regression test usually get written on different timelines, and that split is why coverage lags behind fixes. Braintrust’s guidance on turning production failures into regression tests recommends promoting the failed trace into a versioned dataset row and writing a scorer for the specific failure pattern, then gating releases on that scorer in CI.

Skip that step at fix time, and the default is to ship the patch and defer the eval to a backlog item that competes with the next sprint’s features. The engineer who patches the bug is optimizing for closing the ticket, not for coverage. Writing a grader is a second task with its own definition of done, and it carries no urgency once the original symptom stops. Weeks later, whoever picks up the backlog eval has to reconstruct the failure from a stale ticket description instead of the trace that actually produced it, which is most of why the same bug keeps showing up in postmortems as a repeat entry rather than a new one.

How to Build Agent Regression Test Coverage From a Single Fix

A fix becomes regression coverage only when the grader for it ships in the same commit as the patch, not after. One practitioner’s postmortem framework, Dhiraj Das’s, puts the test plainly: “If the next run can fail the same way without being caught earlier, the postmortem was documentation, not engineering.”

The same standard applies to a routine bug fix, not just a formal incident. If a CI suite cannot tell the difference between the state before the fix and the state after it, the fix is not finished. There is a compounding effect too: each fix-time grader is a statement of what the agent is supposed to do that no upfront spec captured, held outside the agent’s code, so the suite gets sharper with every incident instead of staler.

The mechanics of extracting a failing trace and turning it into a grader are a separate skill, covered in how to turn a production trace into a regression test. That piece assumes someone has already decided to write the eval; this one is about why that decision keeps getting deferred. The person who fixes the bug and the person who would own the eval are rarely working against the same deadline, so the extraction step has to be part of the fix’s definition of done, not a follow-up ticket assigned to whoever has time later. Regression detection that runs against every PR is what enforces that once the habit exists: a fix without a grader can still merge, but it merges without the protection the next deploy needs.

When that grader does fire, the question becomes which change reintroduced the bug. A deploy bundles a prompt edit, a refactor, and a dependency bump, and any of them could be the one. Tessary traces the failing grader back through the agent’s chain to the change responsible, including the causes a PR gate never sees: the gate only fires on changes that ship through a pull request, while a model provider update or a tool returning different data brings the bug back with no deploy at all.

Langfuse and Braintrust run and track the scores you set up, and they tell you when a score moves. Tessary tells you why it moved and which change moved it. Connect a Langfuse or Braintrust source and Tessary ingests the traces you already emit, promotes the failing trace into a regression grader, and runs the suite against every deploy, with no credit card required to start.

Find what broke your agent

Frequently asked questions

What is agent regression test coverage?: A grader written for the exact failure a bug fix addressed, added to the eval suite in the same commit as the patch and run in CI on every future deploy. It differs from general eval coverage, which tests cases the team anticipated. Regression coverage exists specifically because a fix happened, so the eval suite can tell the difference between the state before the fix and the state after it on every subsequent change.
Why do LLM agent bugs come back after being fixed?: An agent bug returns after a fix ships because the fix changed the code or prompt, but no grader was added to check for the specific failure. Without one, a later prompt change, a code refactor, or a model version update that never touches your repo can reintroduce the same behavioral gap in a different form, and nothing is pointed at that failure to flag it.
How do you prevent agent bug recurrence?: Prevent agent bug recurrence by writing the regression test at fix time, not as a follow-up task. Extract the trace that shows the failure, write a grader that checks the failure mode rather than the exact bad output, and add it to the eval suite in the same commit as the fix. Wiring that suite into a CI gate means the next deploy runs the grader automatically, so a reintroduced failure gets caught before it reaches users again.
What causes an agent eval coverage gap?: An agent eval coverage gap happens when the eval suite passes cleanly because it was never built to catch the specific behavior that later broke. Anthropic's own April 2026 postmortem on Claude Code quality issues described running weeks of internal testing with no regressions in their evaluation set, then shipping a change that still caused a measurable quality drop. The suite was accurate for what it covered; the gap was in what it never tested.
When should you write a regression test for an agent bug?: Write the regression test in the same work session as the fix, before closing the ticket. Once the incident's urgency fades, writing the eval competes with new feature work and rarely wins, and the person picking it up later has to reconstruct the failure from a stale description instead of the original trace. Treating the grader as part of the fix, not a follow-up task, is what keeps regression coverage from falling behind.
How is regression coverage different from a broader eval suite?: A broader eval suite covers the inputs a team anticipated when building it. Regression coverage targets a narrower, specific set: the exact behaviors that have already failed in production and been fixed. The two overlap but serve different purposes. A broad suite catches gaps in what you expected; regression coverage guarantees a bug you already paid for once cannot silently reappear on the next change.

Written by

Akhil Varma · Founder, Tessary

Akhil builds Tessary — AI personas that run real-browser usability tests on B2B SaaS products. Previously shipped product at multiple early-stage startups; writes about usability testing, AI personas, and the economics of B2B research.