Is every quality shift after a change a regression?

No. A regression is a drop the team didn’t want, and the exact same movement in a metric can be what a change was meant to do. Swap in a cheaper model on purpose and pass rate on a hard grader might fall a little; that’s an accepted tradeoff, not a regression, if the team chose it knowing the cost. Tighten a refusal policy and a false-refusal rate climbs; that’s the point of the change, not a side effect of it. Statistics alone can’t settle this, because a shift and a regression look identical in the numbers: both are the distribution moving after something shipped. What decides it is knowing what the agent was meant to do after the change, and checking the shift against that intent rather than against the old baseline by itself. The number alone can’t make that call; someone who knows what the change was supposed to do has to.

keep reading

More on this.

Can a production regression happen without a code change?Yes. The model behind a stable API name can change on the provider's side, and a dependency bump or config edit can shift behavior with no prompt touched.answer →Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Everything on regression detectionThe definition, and every question answered under regression detection.concept →

Send us the traces you already emit.

book a demo