Agent Reliability

Grade each tool call, not the reply

A final-output grader grades the response, not the process that produced it. Tessary runs per-call-site graders against individual ToolCall nodes in your production traces, so failures surface at the step where they occur. And when a grader's trend drops, Tessary traces it back to the change that caused it, in your prompt, your code, or the tool itself.

Find what broke your agent See the evals SDK

The research

Tool selection falls as choice grows

Adaline Labs measured tool selection accuracy across agents as the number of available tools increased. At 4 tools, accuracy was 43%. At 51 tools, it was 2%. The degradation is not linear; it collapses. A final-response grader sees none of this because a wrong tool selection can still produce a plausible-looking reply when the agent compensates in later steps.

43%

tool selection accuracy with 4 available tools

tool selection accuracy with 51 available tools

The problem

Per-step errors cascade silently

A parameter error at step 3 propagates through steps 4 and 5 and still produces a syntactically correct final response. The grader marks it passing. The failure surfaces later, in a user-reported broken workflow.

62%

of parameter-level tool errors cascade to a wrong final answer, per a June 2025 arXiv study on tool-integrated LLM agent stability

arXiv, June 2025

38%

arXiv, June 2025

dimensions fail independently at the call site: tool selection, argument correctness, and output handling, each requiring its own grader

Two approaches

Final-response vs per-call-site eval

Final-response graders are not wrong for text-output agents. They are the wrong unit for agents that use tools: the final response is several steps removed from what the agent actually did.

Final-response grader

Grades the output, not the process

Cannot see which tool was called or what arguments were passed
Passes when a wrong tool call resolves to a plausible-looking reply
No signal on argument type errors or missing required fields
Cannot detect when an agent misreads a structured error as success

Tessary

Per-call-site grader

Grades each ToolCall node in isolation

Runs at the exact step where the failure occurred, not after propagation
Flags wrong tool selection before the incorrect result flows downstream
Catches missing fields, wrong types, and incorrectly extracted values
Detects output misinterpretation before the agent acts on corrupt context

How it works

Five steps from trace to cause

Tessary's trace model represents every tool invocation as its own ToolCall node. Graders run against those nodes directly, not against the turn output.

Connect your trace source
Send production traces via OTLP or pull from Langfuse or Braintrust. Every tool invocation lands as its own ToolCall node in the trace. No re-instrumentation required if your agent is already sending traces.
Write per-tool graders
Describe each failure mode in natural language: "agent calling web-search instead of catalog-search for product queries." Tessary compiles it to a classifier that runs on your traces. One grader per tool per failure type gives you granular coverage without overlap.
Run against real traffic
Each ToolCall node in production gets graded, and verdicts accumulate as a trend per grader. Tessary reads that trend rather than any single verdict: an imperfect grader is wrong at a roughly constant rate, so a failure rate that moves signals a real change. Failures link to the exact step where they occurred, not the final response that obscured them.
Gate deploys on tool-call risk
The diff classifier reads the next PR and reports which tool-call graders are at risk, the same attribution engine run against the diff before it merges. A GitHub Action blocks the merge or posts the trouble report as a PR comment, with a per-grader risk score you can threshold by tool criticality.
Trace a failing trend to its cause
A gate only covers changes that ship through a PR. A tool that changes its response schema, an upstream agent whose output shifts, or a provider model update reintroduces failures with no deploy at all. When a tool-call grader trend breaks, Tessary walks the failing dimension back through the chain to the change behind it, in your repo or outside it, and names that change.

What you get

Three call-site failure modes covered

One agent turn can include a dozen tool calls. Each one can fail on any of these three dimensions independently. Tessary runs one grader per tool per failure type.

Tool selection grader

Did the agent invoke the right tool for this step? An agent with 20 available tools that selects the semantically adjacent one passes a final-response eval. A selection grader catches it at the call site, before the wrong result propagates.

Argument correctness grader

Did the agent pass the right parameters? Wrong types, missing required fields, or values incorrectly extracted from a prior step register as silent failures. The tool returns HTTP 200, the agent continues, and downstream steps run on corrupt context.

Output handling grader

Did the agent correctly interpret what the tool returned? An agent that misreads a structured error as a success response, or silently truncates a partial result, carries that corruption into the next call without flagging it.

A worked example

The Tuesday a tool changed underneath the agent

A catalog-search vendor changes its error payload from a structured code to a prose message. No deploy went out and nothing in the agent's repo changed. The agent starts reading those errors as successful empty results and answering that no products were found, which is a well-formed, plausible reply. The final-output grader stays flat. The output-handling grader's trend breaks within hours, and Tessary attributes the break to the tool's response change rather than to a commit, because no change in the repo lines up with the drop.

FAQ

Questions about tool-call evals

Tool-calling agent evals are graders that assess each tool invocation separately: did the agent call the right tool, pass the right parameters, and handle the output correctly? They run against individual ToolCall nodes in an agent trace, not just the final response. This catches failures that look acceptable in the end output but occurred mid-step.

A wrong tool call can cascade through subsequent steps and still resolve to a plausible-looking final response. The grader sees the output and passes it. The error surfaces later when a user reports a broken workflow. Per-call-site graders run at each step, so the failure is caught where it occurs rather than after it has propagated.

Three things: tool selection (did the agent invoke the right tool?), argument correctness (did it pass the right parameters with correct types and values?), and output handling (did it correctly parse and use what the tool returned?). Each check requires its own grader because the failure modes and downstream consequences are different.

Tool-calling evals run as part of the pre-deploy gate. Tessary's diff classifier reads the PR, maps the touched code paths to the tool-call graders most likely to regress, and posts a pre-deploy trouble report. A GitHub Action can block the merge if the risk score for a specific tool's grader exceeds your threshold. The gate only fires on changes that ship through a PR, so the same graders keep running on production traffic to catch the causes that never appear in one.

A per-call-site grader runs on a specific ToolCall node in the agent trace, not on the final session output. It evaluates that one invocation in isolation: correct tool selected, correct parameters passed, correct output interpretation. One agent turn can include ten tool calls; each can have its own grader with its own pass/fail threshold.

Nothing in your diff explains the drop, because nothing in your repo changed. A vendor renames a field, restructures an error payload, or starts paginating a result, and the agent quietly misreads what comes back. The output-handling grader's trend breaks, and Tessary attributes the break to the tool's response change rather than to any commit, since your own change history has nothing that matches the timing.

You can run and track per-step scores in both, and Tessary pulls traces from either without re-instrumentation. The difference is what happens after a score moves. Langfuse and Braintrust tell you a tool-call score dropped; Tessary tells you which change dropped it: a prompt edit, a code change, or the tool itself returning different data.

Keep exploring

Get started

Grade tool calls, not just outputs.

Connect a Langfuse or Braintrust trace source, with no credit card required, and see which tool call is failing and which change made it fail.

Find what broke your agentAlready on Langfuse or Braintrust? Pull a trace slice into Tessary without re-instrumenting.