Agent Reliability

Continuous evaluation, scoped to what changed

Full eval suites on every PR block your merge queue. Skipping evals on routine changes erodes coverage. Tessary reads each PR diff and scopes the eval run to what the change actually touches, and when a grader trend drops in production, traces the regression back to the change that caused it.

Find what broke your agentSee regression detection
The pattern

Coverage drops at high ship frequency

Teams build eval harnesses at launch, then skip them on routine changes because the latency blocks the merge queue. A blanket per-PR rule compounds compute cost with deploy frequency. A blanket skip-on-routine rule erodes coverage until a prompt edit causes a production regression. The Evaluation-Driven Development and Operations paper describes a third path: match evaluation intensity to the risk surface of each change, not to a fixed calendar or a per-PR rule.

20 min

the suite latency where teams start skipping evals to protect the merge queue

4 triggers

for a full suite: new capability, model upgrade, major prompt restructure, or tool definition change
The problem

The scope problem compounds

Blanket rules force a tradeoff between eval latency and coverage. A deliberate scope strategy removes the tradeoff.

3 scopes

coverage layers that match the deployment pipeline: full suite for high-risk changes, targeted gate for incremental changes, production monitoring for live traffic

Evaluation-Driven Development and Operations, 2024

20–50

test cases for a starting regression suite: drawn from real production failures and run on every PR as a CI gate before graduating to full coverage

Anthropic, demystifying evals for AI agents

0 coverage

for failure modes not yet in the test set: pre-deploy evals cannot catch failures outside the fixed dataset. Production monitoring fills this gap on live traffic.

Anthropic, demystifying evals for AI agents
Two approaches

Blanket rule vs. risk-matched scope

A blanket per-PR rule and a risk-matched scope both run evals on every change. The difference is which evals run and why.

Blanket coverage rule

Same scope on every deploy

  • Full suite on every PR: eval latency blocks the merge queue at high deploy frequency
  • Skipping routine changes: coverage drops until a prompt edit regresses in production
  • No signal on which failure families a given change is likely to trigger
  • Compute cost compounds with deploy frequency, creating pressure to reduce suite size
Tessary

Risk-matched eval scope

Scope varies with what the change touches

  • Full suite fires on model upgrades, new capabilities, prompt restructures, and tool definition changes
  • Targeted gate runs for incremental prompt edits, config updates, and code changes outside the agent path
  • Diff classifier identifies failure families before any eval runs, from the PR diff alone
  • Coverage maintained at any deploy frequency without blocking the merge queue
How it works

Four steps to scoped coverage

Tessary connects your existing trace source, builds the regression suite from real failures, and scopes each eval run to the risk surface of the change.

  1. Connect your trace source

    Pull sessions from Langfuse or Braintrust without re-instrumentation. Your existing traces become the source of truth for both pre-deploy evals and production monitoring.

  2. Build a regression suite from production failures

    Start with 20 to 50 test cases drawn from real failures. Tessary drafts them from your traces; you correct them rather than write them from scratch. Run the suite on every PR as a CI gate, and as production monitoring surfaces new failure modes, they become new test cases.

  3. Scope each eval run with the diff classifier

    Each PR gets a comment before any eval runs: the surfaces the diff touches, the failure families those surfaces have triggered before (tool-call errors, context loss, schema mismatches), and whether the change warrants the full suite or the targeted gate.

  4. Close the loop with production attribution

    Graders run over live traffic and catch what the test set cannot anticipate. When a grader trend drops, the regression traces back to the change that introduced it, whether it shipped through a PR or never touched your repo at all.
The distinction that matters
After an agent is launched and optimized, capability evals with high pass rates can 'graduate' to become a regression suite that is run continuously to catch any drift.
AnthropicDemystifying evals for AI agentsanthropic.com/engineering/demystifying-evals-for-ai-agents ↗
What you get

Diff-scoped risk prediction

Risk prediction before the merge, production monitoring and attribution after it. The CI gate blocks or passes on the risk score.

Diff classifier on every PR

A retry-logic change in your tool-calling path flags the tool-selection and error-recovery failure families. A README edit flags nothing. The risk report posts as a PR comment before any eval runs.

CI gate integration

Block a merge when the risk score exceeds the configured threshold. Pass routine changes without triggering a full suite run.

Targeted regression gate

For incremental prompt edits, config changes, and code changes outside the agent path, run only the relevant subset of the regression suite.

Full suite trigger

Model upgrades, new capabilities, major prompt restructures, and tool definition changes trigger the full eval suite automatically.

Production monitoring

Some degradations have no diff at all: an upstream agent's output shifts, or the provider updates the model underneath you. Only production monitoring sees those, and attribution traces them back even though nothing in your repo changed.

Regression suite feedback loop

Every production failure links to the change that introduced it, in or outside your repo, and becomes a candidate for the next regression test case. Connect Langfuse or Braintrust and the loop closes automatically.
FAQ

Questions about continuous evals

Continuous agent evaluation is the practice of running automated quality checks on an LLM agent as part of the development and deployment process. Rather than evaluating at launch only, teams run evals on every change, on a schedule, or in response to production signals. The right approach depends on the risk surface of each change and the cost of running the eval suite.
Running evals on every deploy is the right default when your eval suite is fast and cheap to run. When full suites take 20 minutes or cost significant compute per run, teams typically run targeted regression gates on routine changes and reserve full suites for high-risk deploys such as model upgrades, new capabilities, or major prompt restructures.
Run a full eval suite for new capabilities, model version upgrades, major prompt restructures, or changes to tool definitions. Run a targeted regression gate for incremental prompt edits, config changes, and small code updates where the risk surface is narrow. Anthropic's engineering guide recommends graduating successful capability evals into regression suites that run continuously.
Pre-deploy evals run against a fixed dataset before a change ships. They validate known failure modes and capability thresholds. Production monitoring runs graders against live traffic after a deploy. It catches failures that evals miss: edge cases not in the dataset, non-deterministic behavior, and distribution shifts in real user queries. It also covers causes that never appear in any PR, since a CI gate only fires on changes that ship through one. The two are complementary, not interchangeable.
Running evals in CI validates the agent against a test dataset. Pre-deploy prediction reads the change itself: the diff, the touched surfaces, and the failure families historically associated with those surfaces. It can flag risk before any eval runs, helping teams decide whether to trigger a full suite or a targeted gate. Tessary's diff classifier does this automatically on each PR.
Start with a regression suite of 20 to 50 test cases drawn from real production failures. Tessary drafts the cases and graders from your agent's observed behavior; you review and correct them rather than write them from scratch. Run the suite on every PR as a CI gate, add production monitoring for what the test cases do not cover, and fold new failure modes back into the suite as they surface.
When a grader trend drops, Tessary traces the failing quality dimension back through the agent's chain to the change that caused it: a prompt edit, a tool update, a dependency bump, an upstream agent's shift, or a model provider update that never touched your repo. Detection reads the trend across grader verdicts over time rather than any single score, so an imperfect grader's stable misfire rate does not mask a real jump. Attribution runs on the same traces the eval suite uses.
Keep exploring

Related pages
Get started

Scope every eval to the change.

Connect your Langfuse or Braintrust source and get a diff-scoped risk report on each PR before the merge. They show you a score moved; Tessary tells you which change moved it. Starting is free, no payment details required.

Find what broke your agentNo re-instrumentation required if your agent is already on Langfuse or Braintrust. When a cause hunt hits a gap in the traces, that gap is the next node to instrument.