How do you detect an agent regression after it's already in production?

Compare a window of production runs against a reference window from before the change you suspect, using the same measure of quality on both, such as a grader’s pass rate or the average of a score. A regression is that measure moving by more than the noise you’d expect from comparing two samples of the same underlying population, and it has to hold up over enough runs that one bad stretch can’t explain it. In practice that means picking a call site, defining what “before” means for it, your last release or a rolling week, and re-running the comparison as new traffic arrives rather than checking once and stopping. It also has to run on its own schedule, not only around your deploys, because the model, prompts, and tools an agent depends on can each change without your team shipping anything. A measure holding steady across repeated checks is the evidence that nothing regressed.