Agent Reliability

Grade what your multimodal agent actually saw

An agent that misreads a screenshot can still return an answer that looks right, and a text-only eval will pass it. Tessary grades the read and the action separately on your production traces, watches both trends, and when the read starts getting worse, tells you which change caused it.

Find what broke your agent See the trace model

The research

A correct answer can hide a misread

Agent-X, a benchmark of vision-centric agentic tasks spanning web browsing, security and surveillance, and autonomous driving, found that GPT, Gemini, and Qwen model families all complete fewer than half of these tasks end to end when every step in the chain has to be right. Scoring only the final answer hides where the chain actually broke: did the agent misread the image, or read it correctly and still act on it wrong? Picture an agent that reads an invoice total of $1,240 as $12,400 and flags it for escalation. Every check that only reads the response passes it, because the response is well formed and confidently phrased. Only a grader that checks the read against the image catches the misread.

Agent-X benchmark, arXiv:2505.24876 ↗

Two halves, two graders

Perception and action, graded separately

An agent can pass one and fail the other, and only a grader that sees both catches it.

Reference-based visual grader

Checks perception

Checks what the agent claims it saw against a known reference for what the image contains
Works like a text grader checking a claim against a golden answer, applied to the visual observation
Doesn't see what the agent did next

Tessary

Output-conditioned action grader

Checks the response

Checks whether the downstream tool call or response is consistent with what the agent claimed to see
Catches a correct read that still led to the wrong action
Catches a misread that got lucky on the action anyway

How it works

Four steps from a screenshot to the cause

The image and the text turn already live in one record. Grading uses both halves; attribution follows the trend back to whatever changed.

Ingest the trace whole
The image and the text turn arrive as one record through the OTLP receiver, so nothing about the visual context gets stripped before grading runs.
Grade perception and action separately
A reference-based visual grader scores what the agent read from the image; an output-conditioned action grader scores what it did about it. Both run over the same traces as your text-only graders.
Read the trend, not one verdict
No visual grader is perfect. Tessary reads the trend across verdicts over time rather than trusting any single score; an imperfect grader misfires at a stable rate, so a failure rate that climbs from 3% to 11% signals a real shift in what the agent is seeing. A flagged failure routes to Slack, PagerDuty, or a webhook.
Trace the drop to the change that caused it
When the perception trend drops, Tessary walks it back to the change behind it. Sometimes that is a pull request that touched image handling or a vision prompt, and the same attribution runs against the diff before it merges. Often no PR touched vision code at all: the provider updated the vision model underneath you, or an upstream tool started sending lower-resolution screenshots. Tessary traces the drop to that change either way.

What changes

What separating perception and action buys you

Grading the two halves against each other, not just the final output in isolation.

Catch the misread before the wrong action ships

A grader that only checks the final response can't tell you the agent acted on a misread number in a screenshot. One that checks the visual claim first can, and its trend shows reads that used to be right starting to drift.

One trace holds the image and the decision

No stitching together a screenshot log and a tool-call log after the fact to figure out what happened.

No separate multimodal pipeline to maintain

Vision graders run through the same grading and attribution pipeline as your text-only graders, with the same change lineage behind every verdict.

Questions

Answered, plainly.

Multimodal agent evaluation grades an agent that reads images, screenshots, or diagrams alongside text, checking both whether it read the visual content correctly and whether it acted correctly on what it read. Text-only evals only check the second half, so a misread that still produces a plausible-looking answer passes unnoticed.

A text-only eval sees the agent's final response, not what it looked at to produce it. If an agent misreads a number in a screenshot but still returns a value that looks reasonable, a text grader has no way to know the value came from the wrong place. Grading has to reach back into the trace to the image itself.

A reference-based visual grader checks what the agent claims it saw in an image, screenshot, or document against a known reference for what the image actually contains. It works like a text grader checking a claim against a golden answer, applied to the visual observation instead of the final response.

An output-conditioned action grader checks whether the agent's downstream action, such as a tool call or a written response, is consistent with what it claimed to see in the image. It catches cases where the agent read the image correctly but still took the wrong action, or misread it and got lucky on the action anyway.

No. A trace model that carries images and text as part of the same record lets vision graders run through the same grading and attribution pipeline used for text-only traces. Teams add graders for the vision-specific failure modes they care about instead of standing up a parallel system.

Benchmarks suggest they're common. Agent-X, a vision-centric benchmark spanning web browsing, security, and driving, found that GPT, Gemini, and Qwen model families all complete fewer than half of its multi-step tasks end to end. Production failures tend to be quieter: reads that used to be right start drifting after a provider updates the vision model or an image source changes format, with no error thrown and nothing in the diff.

Because the change often lives outside your repo. Model providers update vision models behind the same API name, and an upstream tool can start sending smaller or reformatted screenshots. Tessary watches the perception grader's trend in production and, when it drops, traces the drop back to that change, so you're not searching a diff that was never going to explain it.

Keep exploring

Get started

Find the change behind the misread

Point one multimodal trace at it, no credit card required. When the visual grader and the action grader disagree, Tessary shows you the miss; when the trend moves, it shows you the change behind it.

Find what broke your agent