Why isn't a single bad run enough to call it a regression?

Because agent output is nondeterministic: the same input can produce a fine answer one time and a bad one the next, with nothing about the agent having changed in between. One failure is consistent with that ordinary variance, so it carries almost no information on its own. What makes something a regression is a shift in the distribution, the share of runs that fail or the average of a quality score moving after a change, not before it. That’s why detection compares two populations of runs rather than reading a single verdict: a cohort from before the suspected change against a cohort from after it, on the same kind of input. This holds even when the judgment behind each verdict is itself imperfect, because an imperfect grader’s error rate stays roughly constant run to run, so a rate that doubles or triples is a real shift even when no one verdict can be trusted alone.

keep reading

More on this.

Can a production regression happen without a code change?Yes. The model behind a stable API name can change on the provider's side, and a dependency bump or config edit can shift behavior with no prompt touched.answer →Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Everything on regression detectionThe definition, and every question answered under regression detection.concept →

Send us the traces you already emit.

book a demo