all answers

Agent reliability

Regression detection

A regression is a drop in agent quality caused by a change. Regression detection is the practice of noticing that drop by comparing the agent's outcome after a change against its outcome before.

Two properties of agents make this harder than in conventional software. First, agent output is non- deterministic, so a single bad run carries little information. The signal is distributional: a failure rate or quality score that shifts after a change ships. Second, the change that causes a regression can come from anywhere in the system around the model. A refactor that changes what context gets assembled, a dependency bump that alters tool output formatting, or a config edit that changes retry behavior can each regress the agent's behavior.

There's also a judgment problem the statistics alone can't settle. Teams change their agents on purpose, and an intended improvement moves the same numbers a defect does. Whether a detected shift counts as a regression depends on what the agent was meant to do after the change.

6 questions

Answered, plainly.

Can a production regression happen without a code change?Yes. The model behind a stable API name can change on the provider's side, and a dependency bump or config edit can shift behavior with no prompt touched.answer →Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Is every quality shift after a change a regression?No. A regression is an unwanted drop; the same movement in a metric is expected, even desired, when the change was made on purpose to produce it.answer →What regressions will a CI gate never catch?Any regression whose cause never shows up in a diff: a model updated behind a stable API, a dependency's behavior shifting, or a downstream service changing its output.answer →Why isn't a single bad run enough to call it a regression?Agent output is nondeterministic, so one bad run can be ordinary variance. A regression is a shift in the rate of bad runs, visible only across many of them.answer →

Send us the traces you already emit.

book a demo