What is a silent failure in an AI agent?

A silent failure is an agent run that completes normally and gets the answer wrong.

The term predates agents. In any system, a silent failure is one with no error signal attached: the process exits 0, the request returns 200, no exception is thrown, and the wrong thing happened anyway. What makes agents produce them constantly is that the output is prose. Fluent text passes every check a status code can perform, and nothing about the way an answer reads tells you whether it is true.

The trace shape is ordinary. A tool returns a record that was correct last week, the agent reads it, summarizes it in a paragraph the user has no reason to doubt, and the span closes with status OK. No layer reported a problem, because none of them was checking the record against anything.

That is why standard monitoring never sees one, and why detection means reading what the agent said.

keep reading

More on this.

How do you detect silent failures in production?By grading what the agent actually did against the intended outcome, on enough production traffic that a rare failure has a real chance of showing up.answer →How many traces do you need before you're likely to catch a rare failure once?The failure rate times the share of sessions you review gives the per-session chance of catching it, so a rare failure takes thousands of sessions to show up once.answer →Why does a trace look normal when a tool returns stale data?Because every span in the trace is individually correct: the tool call succeeded, returned a well-formed payload, and the agent used it exactly as designed.answer →Everything on silent failuresThe definition, and every question answered under silent failures.concept →

Self-host Tessary.

Free and open source. Point it at the traces your agent already emits.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y