What can't a pre-deploy CI gate catch?

A pre-deploy gate only ever judges a diff, so anything that degrades an agent without touching your code walks straight past it. Three real causes do exactly that: a provider swapping the model behind an API you call, an upstream agent or service changing what it returns, and a tool reshaping its own response format.

OpenAI’s April 2025 postmortem on GPT-4o is the clearest public example of a provider-side change like this. An update made the model noticeably more sycophantic, users flagged it within days, and OpenAI rolled it back days later, all inside its own pipeline. The shift landed in ChatGPT with no commit, no version bump, and nothing in any outside team’s repository to review.

A gate built to read diffs structurally can’t see that; it’s a property of what a diff is, not a gap in any one team’s setup. Catching it means watching live traffic for a silent failure instead, since the run itself still looks completely normal.