What regressions will a CI gate never catch?

Any regression whose cause never shows up in a diff. A CI gate runs the modified agent against a set of checks before a change merges, so its whole method depends on there being a change in the branch to run against. A provider updating the model behind a stable API name, a dependency’s behavior shifting on its own release schedule, or a downstream service changing its response format all regress an agent with nothing in any pull request to point the gate at. A CI gate was built to check a change your team is about to ship, so a regression with no diff behind it simply falls outside what that check can see. That’s why production monitoring exists as its own practice, separate from pre-merge testing: it watches what the agent actually does on a rolling basis, and catches a regression the gate never had a diff to run.

keep reading

More on this.

Can a production regression happen without a code change?Yes. The model behind a stable API name can change on the provider's side, and a dependency bump or config edit can shift behavior with no prompt touched.answer →Does a passing unit test rule out a regression?No. A unit test asserts your code executed the right path, such as calling a tool with valid arguments; it says nothing about whether the model's behavior stayed good.answer →How do you detect an agent regression after it's already in production?Compare a window of production runs against a reference window from before the suspected change, on the same quality measure, and watch the gap hold up over enough runs.answer →Everything on regression detectionThe definition, and every question answered under regression detection.concept →

Send us the traces you already emit.

book a demo