Agent Reliability

Predict agent failures before you deploy

When an agent degrades in production, Tessary traces it to the change that caused it. Pre-deploy prediction is that history turned forward: every failure traced to a change becomes a risk signal for the next PR touching the same surface, posted as a trouble report before the merge.

Find what broke your agent
The signal

Most agent errors are not logic changes

Datadog's State of AI Engineering 2026 report found that in February 2026, 5% of all LLM call spans reported an error, and 60% of those errors were exceeded rate limits, nothing in the agent's logic. Agent framework adoption nearly doubled year over year in the same report, so a growing share of the stack sits in dependencies the team did not write. Standard pre-deployment checks treat a framework bump as maintenance, not a risk signal. This is the class of change the diff classifier reads.

Datadog, State of AI Engineering 2026 ↗

5%

of LLM call spans reported an error in February 2026

60%

of those errors were exceeded rate limits, no agent logic involved
The Problem

The gap standard pre-deploy tests leave

Changes that do not read as agent logic pass standard tests untouched, even when they alter what the model receives or how its outputs land in downstream code.

2x

year-over-year growth in agent framework adoption, from over 9% of orgs in early 2025 to almost 18% by early 2026

Datadog, State of AI Engineering 2026

6

surfaces the diff classifier reads on every PR: prompt files, model version pins, library dependencies, tool schemas, call chain structure, and infrastructure config

0

re-instrumentation required if your agent is already sending traces to Langfuse or Braintrust
How It Works

Three components, one sequence

The mechanism runs on the code diff, not on test scenarios. Three components work in sequence to surface production risk before the merge.

  1. Diff classifier reads the PR

    The classifier maps every surface the PR touches: prompt files, model version pins, library dependencies, tool schemas, call chain structure, and infrastructure configurations. Each surface belongs to failure families (context loss, tool hallucination, schema errors) that have historically triggered when that surface changed.

  2. Risk routing scores the PR

    Every production failure Tessary has traced to a change becomes history the risk model reads, with the commit SHA attached when the cause shipped through your repo. When a new PR touches a surface that previously caused a tool hallucination or a context loss failure, that history surfaces in the trouble report.

  3. Trouble report posts on the PR

    Before the merge, the report shows which failure families the change endangers and the confidence level for each. Severity routing tells the CI step whether to block the merge, post a review comment, or pass silently based on the threshold configured per failure family.
Two Approaches

Diff-aware vs scenario-based testing

Standard pre-deploy tests check what engineers expect to break. Tessary reads the diff against your agent's production failure history, and leaves the causes no diff contains to production monitoring and attribution.

Standard pre-deploy tests

Evaluate agent behavior against expected scenarios

  • Pass when agent logic is unchanged, even when tool schemas shift
  • Miss framework upgrades, model version pins, and dependency drift
  • Require engineers to anticipate every failure scenario in advance
  • No link between a specific code change and the failure family it triggers
Tessary

Tessary pre-deploy prediction

Reads the diff, maps to production failure families

  • Reads non-prompt changes that standard tests ignore
  • Maps changed surfaces to failure families from production history via commit-SHA lineage
  • Posts the trouble report before the merge, not after users report failures
  • CI gate configurable per failure family: block, comment, or silent
The Report

What the trouble report shows

The report is built from the graders running on your production traffic in the connected trace source. It reflects your agent's actual failure history, not a generic checklist.

Failure family mapping

Identifies which failure families the specific changes in the PR are most likely to fire: a dependency bump that touches your tool-calling path flags schema errors, a prompt file edit flags output quality.

Confidence-level breakdown

Shows high-, medium-, and below-threshold signals so teams can calibrate the CI gate in report-only mode before switching to a blocking gate.

Change lineage

Each risk signal ties back to the change that caused the original production failure, with the commit SHA attached when that change shipped through your repo, so engineers can trace where every signal came from.

Threshold calibration from history

Thresholds start in report-only mode. A few deploy cycles of reports show which families fire noisily and which predict real failures, and the blocking gate inherits the levels that held up.

Production-grounded history

The report reflects your agent's actual production history from the connected trace source, not a generic failure checklist. Common families include context loss, tool hallucination, schema errors, and output quality drops.

OTLP and native integrations

Ingest production traces via an OTLP receiver (gRPC 4317 or HTTP 4318), or connect Langfuse or Braintrust as a pull source without re-instrumenting your agent.
Why tests alone miss it
Production traces reveal failure modes you couldn't have predicted and help you understand what 'correct behavior' actually looks like for real user interactions.
LangChainAgent Observability Powers Agent Evaluationlangchain.com/blog/agent-observability-powers-agent-evaluation ↗
FAQ

Questions about pre-deploy prediction

Pre-deploy agent failure prediction reads the code changes in a PR, maps those changes to the production failure families they are statistically likely to cause, and posts a trouble report before the merge. A CI gate can block the deploy if the risk score exceeds a threshold. The result is that production regressions surface at review time, not after users report them.
The diff classifier reads every surface in the PR: prompt files, model version pin updates, library dependency changes, tool response schemas, call chain structure, and infrastructure configuration. These non-prompt changes are the class that standard pre-deployment tests miss because they do not register as changes to agent logic, even though they alter what the model receives or how its outputs are consumed.
Risk routing maps each surface in the PR diff to the failure families that surface has historically triggered in production, using commit-SHA lineage where the original cause shipped through your repo. When the same library, schema, or configuration that previously caused a tool hallucination appears in a new PR, that history lands in the pre-deploy trouble report.
A GitHub Action reads the trouble report and compares each risk score to your thresholds. Teams typically start in report-only mode, watch a few deploy cycles to see which reports proved accurate, then let the Action hold merges for the failure families that earned it.
No. If your agent is already instrumented on Langfuse or Braintrust, connect it as an upstream source and Tessary pulls a slice on demand. If you send OpenTelemetry traces directly, point the OTLP receiver at Tessary's collector endpoint. Langfuse and Braintrust hold your traces and track the scores; Tessary reads that same history to say which change moved a score, and the pre-deploy report is built from those answers. When a cause hunt hits a gap in the traces, that gap is the next node to instrument, and the next report gets sharper.
Failure families are the classes of production failure your graders score on live traffic: context loss, tool hallucination, constraint violations, schema errors in tool responses, and output quality drops are common ones. When a grader trend breaks in production and Tessary traces it to a change, that family becomes a risk signal for future PRs touching the same surface. The report reflects your agent's actual production history, not a generic checklist.
Some causes never appear in a PR. A provider model update, an upstream agent's shift, or a tool returning different data reintroduces a failure with no deploy at all, so no CI gate can fire on it. Tessary's production monitoring reads the trend across grader verdicts (not any single score, so an imperfect grader's stable misfire rate does not hide a real jump) and traces a drop back to the change that caused it, whether or not that change touched your repo. Pre-deploy prediction covers the changes you ship yourself; production attribution covers the rest.
Keep exploring

Related pages
Get started

Run the check on your next PR.

Point Tessary at your repository and connect your trace source. The next trouble report shows which failure families your PR touches, and when something slips through anyway, the same history says what caused it. Connecting is free, no card required.

Start with your repoNo re-instrumentation required if your agent is already on Langfuse or Braintrust.