How do I find out which change caused my agent's quality drop?

The cause is whichever candidate change, tested against the failing traces, is the only one that explains them: timing matches, the failure pattern matches what that kind of change would cause, and reverting it, even hypothetically, accounts for both the traces that failed and the ones that didn’t. To get candidates worth testing, pull every trace where the drop shows up into one cohort first, since a single bad trace can’t tell you what changed, only a pattern can. Find when that cohort’s failures start, then list every candidate that touched the system around that time: a code commit, a prompt edit, a model swap, a tool whose behavior shifted, or an upstream system the agent depends on. Test each one against the traces, not against how plausible it sounds, and drop whatever doesn’t hold. If the failing traces share one code path, the candidate list narrows fast, because a change has to have touched that path to be a suspect at all.

keep reading

More on this.

How do I tell which deploy was live when a session failed?Match the session's timestamp against your deploy or commit log; the version whose window contains that timestamp is the one that produced it.answer →What evidence should a regression investigation include?The named cause, the failing traces and findings that support it, and the hypotheses you checked and ruled out, not just a summary of what changed.answer →What comes back when the cause of a failure cannot be established?Sometimes nothing does. An honest attribution says no change explains the failure rather than naming the closest guess, and it keeps what it ruled out.answer →Everything on cause attributionThe definition, and every question answered under cause attribution.concept →

Send us the traces you already emit.

book a demo