Agent reliability

Silent failures

A silent failure is an agent run that completes normally and produces a wrong result. The run returns a success status, throws no exception, and the response reads fluently, but the user didn't get what they asked for. The agent misread a tool result, claimed a task was done when it wasn't, answered from stale data, or drifted from its instructions partway through.

The failure is silent because standard monitoring measures properties of the run, and the problem is in the content of the response. Error rates, latency, and uptime all stay healthy while it happens. The first person to notice is usually a user, and a user report arrives some time after the failure started, so by then it has usually happened many times.

Silent failures are typically low-probability, and that determines how long they survive. A failure that hits 2% of sessions, observed through a 5% review sample, surfaces in about one session in a thousand, so evidence accumulates slowly. A single occurrence is also hard to distinguish from noise: one odd trace could be anything, and the pattern only becomes visible once many traces have failed the same way.

6 questions

Answered, plainly.

How do you detect silent failures in production?By grading what the agent actually did against the intended outcome, on enough production traffic that a rare failure has a real chance of showing up.answer →How many traces do you need before you're likely to catch a rare failure once?The failure rate times the share of sessions you review gives the per-session chance of catching it, so a rare failure takes thousands of sessions to show up once.answer →What is a silent failure in an AI agent?A run that completes with no error signal and still gets the answer wrong: exit 0, a 200, no exception, fluent text, and the user didn't get what they asked for.answer →Why does a trace look normal when a tool returns stale data?Because every span in the trace is individually correct: the tool call succeeded, returned a well-formed payload, and the agent used it exactly as designed.answer →Why doesn't monitoring catch a silent agent failure?Standard monitoring watches whether a run completed, not what it said: error rate, latency, and uptime all read healthy on a run that returned the wrong answer.answer →Would stratified sampling fix a silent failure?Only partly. It raises your odds inside a slice you deliberately target, but a failure with no known pattern still hides in whichever slice you left thin.answer →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization

traces: 10,000 per calendar month
stored trace data: 1 GB
retention: 30 days
model credit: $10, one-time, for triage and root-cause analysis
credit card: not required

start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

set up with a coding agentinstall manually

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y