Can a production regression happen without a code change?

Yes. A regression is a drop caused by a change, and that change doesn’t have to be one your team made. Providers update the model behind a stable API name on their own schedule, sometimes without announcing it, so a call site can get measurably worse between two mornings with an empty diff on your side. A dependency bump can reformat what a tool returns, and a config default can shift retry or timeout behavior, neither touching a prompt. In August 2025 a routing bug misdirected some of Anthropic’s own Sonnet 4 traffic, moving from 0.8% of requests on the day it started to 16% in the worst hour weeks later, and it ran that long partly because the evaluations running at the time didn’t catch it. Watch what the agent actually does, not just what your team shipped. The cause of a drop can sit entirely outside your repository.

sources

keep reading

More on this.

Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Is every quality shift after a change a regression?No. A regression is an unwanted drop; the same movement in a metric is expected, even desired, when the change was made on purpose to produce it.answer →Everything on regression detectionThe definition, and every question answered under regression detection.concept →

Send us the traces you already emit.

book a demo