How do I find out which change caused my agent's quality drop?

The cause is whichever candidate change, tested against the failing traces, is the only one that explains them: timing matches, the failure pattern matches what that kind of change would cause, and reverting it, even hypothetically, accounts for both the traces that failed and the ones that didn’t. To get candidates worth testing, pull every trace where the drop shows up into one cohort first, since a single bad trace can’t tell you what changed, only a pattern can. Find when that cohort’s failures start, then list every candidate that touched the system around that time: a code commit, a prompt edit, a model swap, a tool whose behavior shifted, or an upstream system the agent depends on. Test each one against the traces, not against how plausible it sounds, and drop whatever doesn’t hold. If the failing traces share one code path, the candidate list narrows fast, because a change has to have touched that path to be a suspect at all.

keep reading

More on this.

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y