Can a production regression happen without a code change?

Yes. A regression is a drop caused by a change, and that change doesn’t have to be one your team made. Providers update the model behind a stable API name on their own schedule, sometimes without announcing it, so a call site can get measurably worse between two mornings with an empty diff on your side. A dependency bump can reformat what a tool returns, and a config default can shift retry or timeout behavior, neither touching a prompt. In August 2025 a routing bug misdirected some of Anthropic’s own Sonnet 4 traffic, moving from 0.8% of requests on the day it started to 16% in the worst hour weeks later, and it ran that long partly because the evaluations running at the time didn’t catch it. Watch what the agent actually does, not just what your team shipped. The cause of a drop can sit entirely outside your repository.

sources

keep reading

More on this.

Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do I detect agent regressions before users complain?Grade every production turn against the call site's own baseline and alert on a shift in the rate, not on one bad run, so the evidence arrives from traces rather than tickets.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Everything on regression detectionThe definition, and every question answered under regression detection.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y