Agent Hallucination Monitoring: How to Catch Plausible Wrong Output Before Users Do

A June 2026 arXiv preprint studying a production LLM agent runtime found that 70% of its silent failures surfaced only through user reports, not automated monitoring. The runtime had 4,286 tests, 827 governance checks, and a 19-point preflight running, all returning green during most of the incidents studied. Agent hallucination monitoring is what was missing: graders designed specifically for what the paper calls fail-plausible failures, where the model converts corrupted or stale context into confident, fluent wrong output.

Why Error Rates Don’t Catch Agent Hallucinations

The standard monitoring stack catches a specific class of agent failure: exceptions, timeouts, empty responses, and spans that never close. None of these fire on a fail-plausible response. The agent returns HTTP 200, latency is normal, and the output is fluent.

The paper classifies five silent failure types by mechanism. Four produce detectable infrastructure signals: environment mismatches that throw exceptions, design assumption violations that produce empty responses, error-swallowing that generates anomalous logs, and operational omissions that create identifiable missing-step patterns. The fifth type produces no infrastructure signal. The model converts the failure into a well-formed response. Error rates stay flat and exception logs stay quiet. The only divergence is between what the output says and what is actually true.

What Causes Confident Wrong Output in LLM Agents

Three mechanisms account for most fail-plausible failures in production agents.

Conversation history grows across turns. When context window pressure forces truncation, the model loses the earliest turns. A user established in turn 2 that their company does not operate in the EU. By turn 15, that constraint is no longer in the context. The agent generates a response referencing EU data requirements as if they apply. The agent returns HTTP 200 with a fluent, well-formed reply. It is wrong for this user.

A RAG index that is not updated after a policy change presents a related problem. The agent fetches three documents that describe the previous policy and synthesizes them into a confident answer. The output is faithful to the retrieved documents. The retrieved documents describe policy that no longer applies. A faithfulness grader that checks output against retrieved text passes. A grader checking against a current knowledge base does not.

Model version bumps shift how the model weights constraints in the system prompt. The change can be subtle: a safety check the model was respecting in the vast majority of cases starts being skipped on edge inputs after the bump. No test caught it because the eval suite was calibrated to the previous model version’s behavior. This mechanism is documented well beyond one paper: a Stanford and UC Berkeley study measured GPT-4’s rate of generating directly executable code dropping from 52% to 10% between March and June 2023, under the same model identifier.

What Agent Hallucination Monitoring Actually Needs to Check

The graders that catch fail-plausible failures check different things than the graders that catch format errors or missing tool calls.

Faithfulness graders compare the agent’s output against what the retrieval layer actually returned. If the agent claims something the retrieved documents do not support, that mismatch is the signal. A faithfulness grader fires here even when all infrastructure metrics show nothing.

Groundedness graders check claims against a versioned knowledge base, not just against the documents the retrieval layer returned in this session. A retrieval-faithful agent reporting stale policy passes the faithfulness grader and fails the groundedness grader. The distinction matters because the stale-retrieval failure case passes most faithfulness checks.

Context-presence graders run at the observation level, one layer above the raw span. They check whether the expected context was actually in the model’s input before the call. If a required field is missing from the context payload, the grader fires before the model has a chance to substitute a plausible-sounding value.

Session-level graders catch multi-turn contradictions. Did the agent’s response in turn 15 contradict a fact the user established in turn 2? A single-turn grader cannot see this. A grader running across the full session history can.

The pattern of which graders fire is also a fingerprint of the cause. Groundedness failing while faithfulness passes points at a stale index. A context-presence grader firing points at truncation, and a trend break that lines up with a version bump points at the model. Detection tells you a quality dimension broke; the fingerprint is the start of finding which change broke it.

Why Hallucination Graders Need to See the Full Session

Most observability tools surface spans: one LLM call, one input, one output. Fail-plausible failures are often visible only across the session. The constraint the agent violated in turn 15 does not appear in the span from turn 15. It appears in the transcript from turn 2.

Graders that express session-level rules need access to the full Session→Turn→Trace→Observation model. A rule like “did the agent contradict an earlier user-provided fact?” requires reading the session history while scoring a specific turn. A rule like “did the agent’s claim in this turn conflict with the retrieved document from two turns ago?” requires both the current trace and the retrieval result from a prior turn.

Tessary runs hallucination graders directly against the full session model, so a grader scoring turn 14 can read what the user said in turn 2. When a grader fires, the whole trace comes with it. Patterns across sessions surface before the user-report pattern emerges: this grader went from 3% to 11% across sessions after yesterday’s deploy. Individual verdicts from a hallucination grader are noisy, but the trend is not. A grader that misfires at a stable rate keeps misfiring at that rate next week, so a jump of that size signals a real change even with an imperfect judge.

Tessary automates the step after the spike: it traces the fired dimension back through the agent’s chain to the change that caused it, the retrieval index refresh, the model version bump, or the prompt edit, rather than leaving the team to eyeball yesterday’s deploy, which bundled all three. Setting up monitoring and alerts on these graders tracks the fire-rate trend across deploys, so an index update or version bump that shifts failure rates shows up as a trend break before the next customer complaint.

Pull your Langfuse or Braintrust traces into Tessary and it drafts faithfulness, groundedness, and session-coherence graders from your actual production sessions. You correct the drafts rather than write graders from scratch, and the first results appear on real traffic without a credit card. Langfuse and Braintrust hold the traces and track the scores; when a score moves, Tessary tells you which change moved it.

Find what broke your agent