What is a silent LLM agent failure?

A silent LLM agent failure is a production error that does not produce an HTTP error response, exception, or monitoring alert. Infrastructure metrics remain healthy (latency normal, error rate stable, throughput unchanged), but the output is wrong, degraded, or semantically incorrect. Common examples include a tool call that invokes a nonexistent function, a tool that returns a success status with wrong data, gradual output quality decay after a model version update, and rate-limit retries that mask a degraded response. These failures surface as user complaints, not dashboard alerts.

What causes output quality decay in LLM agents?

Output quality decay occurs when agent behavior drifts without a code change triggering an explicit failure. Common causes include model version updates (the same model identifier returning different behavior after a provider update), changes in the orchestration layer that alter context passed to the model at call time, and library updates that affect tool schemas or prompt construction. A 2023 study from Stanford and UC Berkeley found GPT-4's rate of generating directly executable code dropped from 52% to 10% between March and June 2023 with no infrastructure alert firing during that period.

What are hallucinated tool calls in production agents?

A hallucinated tool call occurs when an LLM agent invokes a tool that does not exist or corrupts the tool's required parameters, while the infrastructure returns an HTTP 200 response. A 2025 survey of LLM agent hallucination modes identifies two subtypes: Tool Selection Hallucinations, where the agent selects a nonexistent or irrelevant tool, and Tool Calling Hallucinations, where it selects the right tool but fabricates or omits parameters. Both produce successful infrastructure traces with semantically incorrect behavior.

How do rate-limit cascades cause silent failures?

When an LLM provider returns a rate-limit error, agent frameworks typically retry the request automatically. If the retry succeeds, the infrastructure logs a success and the event appears resolved. But the retry may have hit a fallback model with lower capability, a truncated context window that dropped relevant history, or a degraded response path under load. The Datadog State of AI Engineering 2026 report found that 60% of LLM call errors in February 2026 were rate-limit errors, making this the most common failure class in production agents.

What signals detect silent LLM agent failures in production?

The graders that detect silent failures operate over production traces rather than watching HTTP status codes. A grader defined in plain language describes the failure pattern and runs against real spans. Tool-error auto-extraction catches cases where a tool returned a success status with wrong content. Pre-deploy analysis reads the code diff before a merge and flags which failure families the change is likely to activate, such as tool-call errors, hallucinated citations, or refusal spikes. When a failure is detected, commit-SHA lineage ties it back to the causing commit where one exists. When no commit correlates, as with a provider model update or an upstream tool change, attribution traces back through the agent's spans to the input or dependency that shifted.

All articles

agent reliabilityLLM observabilityproduction monitoring

How to Detect Silent LLM Agent Failures in Production

By Akhil Varma · June 13, 2026

Short answer

Silent failures in LLM agents are production errors that don't produce an HTTP error response or exception. The four main classes are hallucinated tool calls, silent tool errors, output quality decay, and rate-limit cascades. Catching them requires graders over real traffic, not raw trace counts, and fixing them requires tracing a firing grader back to the change that caused it, which is often a change that never touched the repo.

In August 2025, a routing misconfiguration in Anthropic’s infrastructure began directing Claude Sonnet 4 requests to servers configured for 1M-token context windows. The bug affected 0.8% of Sonnet 4 requests on August 5 and, at the worst-impacted hour on August 31, 16% of them. Anthropic’s postmortem is direct about why detection took so long: “More fundamentally, we relied too heavily on noisy evaluations.” Some users saw degraded responses for over a month. This pattern is the defining characteristic of silent LLM agent failures: infrastructure metrics stay normal while output quality degrades.

The silent LLM agent failures that standard monitoring misses

Traces record HTTP status codes, token counts, latency, and the structural shape of tool calls. They don’t record whether a tool invocation produced a semantically correct result, whether output quality degraded compared to last week’s baseline, or whether a successful retry masked a worse response than the original request would have delivered.

The Anthropic incident illustrates the structural problem. Engineers running Claude in production saw no signal from their observability stack. The monitoring infrastructure was watching what monitoring infrastructure watches: latency, error rate, throughput. The behavioral regression, which at the worst-impacted hour affected one in six Sonnet 4 requests, was invisible to those systems.

Hallucinated tool calls and silent tool errors

Tool hallucination in production agents takes two forms, as identified in a 2025 survey of LLM agent hallucination modes: Tool Selection Hallucinations, where the agent invokes a nonexistent or irrelevant tool; and Tool Calling Hallucinations, where the agent selects the right tool but fabricates or omits required parameters. Both types produce HTTP 200 responses with incorrect behavior. Neither triggers an error alert.

Silent tool errors are a separate failure class. The tool runs, returns a success status code, but delivers wrong or incomplete data. The agent receives this response, treats it as valid, and incorporates the error into its output. A trace showing a 200 status on that tool call gives no indication anything went wrong.

Output quality decay and rate-limit cascades

A Stanford and UC Berkeley study on GPT-4 behavioral drift found that directly executable code generation dropped from 52% to 10% between March and June 2023. No infrastructure alert fired during that period, even though the behavioral change across that same model identifier was 42 percentage points on the same task.

Rate-limit cascades follow a similar path at the infrastructure layer. The Datadog State of AI Engineering 2026 report documented that 5% of LLM call spans reported an error in February 2026, and 60% of those errors were caused by rate limits. Agent frameworks typically retry on rate-limit errors. When the retry succeeds, the infrastructure logs a 200. What it doesn’t log is whether the retry hit a fallback model, a truncated context window, or a degraded response path. The count of resolved retries stays on the success side of the dashboard, but the quality of what each retry returned is not recorded there.

Signal types that catch each failure class

The graders that detect these failures operate differently from infrastructure monitoring. A grader defined in plain language runs over production spans and flags calls that match a semantic failure pattern, not an HTTP status code. Tool-error auto-extraction identifies the cases where a tool returned a success status but the content was wrong. Each matching span produces a recorded verdict tied to that specific call, so a firing grader arrives with the exact production traces that triggered it.

Detection reads the trend across those verdicts over time, not any single one. An imperfect grader misfires at a stable rate, so a rising fire rate signals a real change even when individual verdicts are noisy. That is the property Anthropic’s evaluations lacked by its own account: at 0.8% of affected requests on August 5, the degraded output pattern was already present in enough production spans for a trend to move, weeks before the worst hour.

A firing grader also tells you which dimension of quality broke, and that is where cause-finding starts. Where a commit correlates, each verdict carries the commit that introduced it, and the investigation begins at that code boundary. Many causes never appear in a diff, though: a provider updating the model underneath you, an upstream agent’s output shifting, a tool starting to return differently shaped data. For those, Tessary traces the failing dimension back through the agent’s chain to the input that fed it, as far as the traces reach. That is the path that names behavior drift like the Anthropic case, where no commit in any customer repo explains the change.

Pre-deploy analysis is the same attribution engine run before a merge: it reads the PR diff and flags which failure classes the changed code paths are most likely to produce. It covers the causes that ship through a PR. A CI gate alone cannot cover the rest, because a provider model update or a tool returning different data reintroduces a failure with no deploy at all.

Connecting signal detection to your existing traces

Tessary runs these graders on your Langfuse or Braintrust traces without any instrumentation change to your agent. Langfuse and Braintrust will show you that a score moved; Tessary’s job on top of the same traces is to say which change moved it. Connect the trace source, describe the failure mode you’re watching for in plain language, and graders run over production spans from that point on. Add the GitHub Action and the same analysis reads your next PR diff before it ships.

Getting started requires no new instrumentation in your agent code and no credit card. When a cause hunt runs into a gap in the traces, that gap marks the next node worth instrumenting, and the next degradation is easier to trace. Detection tells you something broke. The point is knowing which change broke it, in time to fix it before it does damage.

Find what broke your agent

Frequently asked questions

What is a silent LLM agent failure?: A silent LLM agent failure is a production error that does not produce an HTTP error response, exception, or monitoring alert. Infrastructure metrics remain healthy (latency normal, error rate stable, throughput unchanged), but the output is wrong, degraded, or semantically incorrect. Common examples include a tool call that invokes a nonexistent function, a tool that returns a success status with wrong data, gradual output quality decay after a model version update, and rate-limit retries that mask a degraded response. These failures surface as user complaints, not dashboard alerts.
Why do traces fail to detect silent agent failures?: Traces record infrastructure events: HTTP status codes, token counts, latency, and the shape of tool calls. They don't record whether a tool invocation produced a semantically correct result, whether output quality degraded compared to last week's baseline, or whether a successful retry replaced a failed request with a worse response. Detecting these failures requires graders that evaluate behavior over production spans, not just counts of HTTP 200s and 500s.
What causes output quality decay in LLM agents?: Output quality decay occurs when agent behavior drifts without a code change triggering an explicit failure. Common causes include model version updates (the same model identifier returning different behavior after a provider update), changes in the orchestration layer that alter context passed to the model at call time, and library updates that affect tool schemas or prompt construction. A 2023 study from Stanford and UC Berkeley found GPT-4's rate of generating directly executable code dropped from 52% to 10% between March and June 2023 with no infrastructure alert firing during that period.
What are hallucinated tool calls in production agents?: A hallucinated tool call occurs when an LLM agent invokes a tool that does not exist or corrupts the tool's required parameters, while the infrastructure returns an HTTP 200 response. A 2025 survey of LLM agent hallucination modes identifies two subtypes: Tool Selection Hallucinations, where the agent selects a nonexistent or irrelevant tool, and Tool Calling Hallucinations, where it selects the right tool but fabricates or omits parameters. Both produce successful infrastructure traces with semantically incorrect behavior.
How do rate-limit cascades cause silent failures?: When an LLM provider returns a rate-limit error, agent frameworks typically retry the request automatically. If the retry succeeds, the infrastructure logs a success and the event appears resolved. But the retry may have hit a fallback model with lower capability, a truncated context window that dropped relevant history, or a degraded response path under load. The Datadog State of AI Engineering 2026 report found that 60% of LLM call errors in February 2026 were rate-limit errors, making this the most common failure class in production agents.
What signals detect silent LLM agent failures in production?: The graders that detect silent failures operate over production traces rather than watching HTTP status codes. A grader defined in plain language describes the failure pattern and runs against real spans. Tool-error auto-extraction catches cases where a tool returned a success status with wrong content. Pre-deploy analysis reads the code diff before a merge and flags which failure families the change is likely to activate, such as tool-call errors, hallucinated citations, or refusal spikes. When a failure is detected, commit-SHA lineage ties it back to the causing commit where one exists. When no commit correlates, as with a provider model update or an upstream tool change, attribution traces back through the agent's spans to the input or dependency that shifted.

Written by

Akhil Varma · Founder, Tessary

Akhil builds Tessary — AI personas that run real-browser usability tests on B2B SaaS products. Previously shipped product at multiple early-stage startups; writes about usability testing, AI personas, and the economics of B2B research.