What is the eval coverage gap in LLM agent evaluation?

The eval coverage gap is the difference between the input distribution your evaluation suite covers and what users actually send in production. An agent can pass 95% of its eval suite and still encounter production failures when the suite was built from cases the team anticipated rather than sampled from real traffic.

Why does a high eval pass rate coexist with production failures?

A high eval pass rate and production failures coexist when graders are accurate but scoped to the wrong distribution. If the eval set was built from inputs the team imagined, graders pass those cases while failing inputs real users send. The score accurately reflects the eval distribution, which doesn't match the production distribution.

How do I detect a coverage gap in my eval suite?

Sample 200-300 production spans from your trace source, cluster by intent or topic, and check which clusters appear in your eval set. Topics with no representation are coverage gaps. A second signal: a grader score that drops after a deploy where the failing cases don't appear in the eval set indicates inputs the suite wasn't covering. Coverage analysis tells you where the graders were blind; tracing the drop to the change that caused it is a separate attribution step.

What is the difference between a coverage gap and dataset drift?

Dataset drift is when an eval set goes stale as the product changes over time. A coverage gap is a mismatch between the eval set's input distribution and the production distribution, and it can exist even in a freshly built suite when the inputs were written from expectations instead of drawn from live traffic.

How do I close a coverage gap in my agent eval suite?

Close the gap by sampling grader inputs from production traces rather than writing them from scratch. Pull a slice from your trace source, cluster recent spans by intent, and generate graders for the clusters your current suite doesn't reach. Cases from production traffic cover what users are sending; manually added cases cover known regression scenarios.

Eval Coverage Gap: Why Passing Scores Don't Mean Passing Users

ReliabilityBench tested two agent architectures across 1,280 production-like episodes. Under clean, controlled inputs, success rates reached 96.9%. Under modest perturbations at ε=0.2, which model the semantic variation real users introduce through rephrasing and partial context, rates fell to 88.1%. The architectures and models were unchanged between conditions.

The issue is not that the controlled eval was wrong, but that it covered a narrower distribution than production serves. The eval coverage gap is what separates those two numbers.

Why the Eval Coverage Gap Produces High Scores and Real Failures

The gap is the distance between the input distribution your graders cover and the one users actually send. An agent can pass 95% of its eval suite and still fail in production when the suite was written from a whiteboard, not from traffic.

AlphaEval (April 2026) benchmarked seven production agent deployments and found a consistent pattern: existing evaluation frameworks diverge from production environments in their input characteristics. Production inputs carry implicit constraints, heterogeneous document types, and domain-specific expectations that standard benchmarks rarely include. The audit traced the divergence to how eval sets are built: from cases teams expect to see, not from the traffic they serve.

Why Teams Build Eval Sets That Miss Production Inputs

Most eval suites are written before an agent accumulates production traffic. Teams write inputs based on what they expect users to do, test against those cases, and ship. What makes it into the golden set are the cases the team thought to include.

After launch, users send inputs the team didn’t anticipate: phrasings the agent handles poorly, task types that fall just outside the designed scope, combinations of context and question the prompt was never tested against. The result is a recognizable failure pattern: an agent passes the entire eval suite, then a support ticket arrives weeks later for an input type the graders never touched. Anthropic’s engineering guidance on evals for agents recommends starting with 20-50 cases drawn from real failures rather than hypothetical inputs, and growing the set by sampling production traffic as the agent accumulates users.

Eval sets written this way produce graders that are accurate within a defined range. That range just doesn’t cover what users are actually sending.

The AlphaEval audit found that requirements in real production environments “contain implicit constraints” that don’t appear in conventionally curated evaluation sets. Users don’t describe tasks the way engineering teams imagine they will. They include context that wasn’t in the brief, omit context the team assumed would be there, and combine tasks in ways the eval set never modeled.

How to Detect a Coverage Gap in Your Eval Suite

A coverage gap shows a specific pattern: eval scores stay high while user-facing failures accumulate. The graders are accurate within their scope; the scope doesn’t match what’s happening.

Two approaches surface the gap. The first is direct comparison: pull a sample of 200-300 production spans from your trace source, cluster by topic or intent, and check which clusters have representation in the eval set. Embed the span inputs and cluster them, or have an LLM assign each span an intent label; at 200-300 spans, either takes under an hour. Clusters with no corresponding test cases are the gaps.

The second is grader failure correlation: when a score drops after a deploy but the failing cases don’t appear in the current eval set, the drop is accurate within the eval distribution, but the suite wasn’t covering the inputs that caused it.

A drop like that also raises the question coverage analysis can’t answer: which change caused it. A release bundles a prompt edit, a tool update, and a dependency bump, and some causes never show up in a diff at all, like a model provider update or a tool that starts returning different data. Coverage tells you where your graders are blind. Attribution traces the failing quality dimension back through the agent’s chain to the change responsible, and Tessary runs both against the same traces.

Running graders against production traces rather than a static golden set has a structural advantage here. The test distribution includes what users are actually sending, so coverage reflects production behavior rather than prior assumptions.

What Calibrated Eval Coverage Looks Like Against Real Traffic

An eval suite with good coverage has cases that came from what users sent, not from what the team expected users to send. The failure modes in the graders match failure modes the agent actually produced. Grader calibration reflects production spans, not anticipated outputs.

The practical version: pull a slice of traces from your Langfuse or Braintrust source, cluster the inputs by intent, flag the recurring failure patterns, and generate graders for the clusters your current suite doesn’t reach. Langfuse and Braintrust will show you the scores and the traces; they won’t tell you which input clusters your graders have never seen, or why a score moved. That analysis is the layer Tessary adds on top of the traces you already keep.

Tessary does this from your existing connector: it pulls production spans, clusters them by input pattern, and flags the clusters your current graders haven’t reached. The graders it proposes are drafted from the agent’s observed behavior; you correct them rather than write them from scratch, and the intent they encode lives outside the agent’s code, getting sharper as bugs are fixed. Manually added cases cover known regression scenarios; traffic-derived cases cover what users are actually sending, and together they produce a test distribution that reflects production behavior. Evals that evolve with the product close the coverage gap by design, because the inputs come from the same source as production behavior.

Your eval suite already covers the cases you wrote. Connect your trace source to see the input clusters your graders have never touched, and when a score drops, the change behind it. The first report doesn’t need a credit card.

Find what broke your agent