Agent Reliability

Your fallback model can fail without an error

Multi-provider agent evals grade a fallback response with the same graders used on the primary model, regardless of which vendor produced it. Routing gateways like OpenRouter only detect errors and timeouts, not a response that returns successfully but performs worse than the model it replaced. When fallback quality does drop, Tessary traces it to the change that caused it, whether that change shipped in your repo or on the provider's side.

Find what broke your agentSee the trace model
The problem

Routing gateways don't grade quality

A routing gateway's failover reacts to errors and timeouts. None of it measures whether the response that came back is as good as the one it replaced, and a degraded fallback can serve traffic for weeks before anyone connects a metric dip to the routing change. Tessary flags the drop as the trend moves, not after the damage is done.

5

elevated-error incidents on individual Claude models between July 4 and July 13, 2026, each a window a cross-provider fallback chain would activate

Anthropic status page

5%

of all LLM call spans reported an error in February 2026, measured across production traffic

Datadog, State of AI Engineering 2026

60%

of those errors were exceeded rate limits, exactly the condition that flips traffic onto a fallback model

Datadog, State of AI Engineering 2026
The research

Judge model choice can flip the grade

Grading a fallback response consistently is its own problem. A 2026 UC Berkeley evaluation of LLM-as-judge models found that judge rankings shift by four or more positions across benchmarks for more than half the models tested, and position bias between the least and most biased judge spans nearly two orders of magnitude. Pick the wrong judge model to grade fallback traffic, and a single grade can't be trusted. That is why Tessary reads the trend across grader verdicts over time rather than any one score: an imperfect judge is wrong at a fairly steady rate, so a moving trend still signals a real change in the fallback's behavior.

Reliability without Validity ↗

4+

position shifts in judge rankings across benchmarks, for more than half the models tested

~100x

spread in position bias between the least and most biased judge
The gap

What a router checks vs. Tessary

A routing gateway's checks stop at whether the call succeeded. Multi-provider agent evals check whether the response itself held up.

TessaryRouting gateway
Detects a provider error or timeoutYes, surfaced from the trace; your gateway still handles the failover itselfYes, this is what a routing gateway is built for
Detects a rate-limited or down providerYesYes
Checks the fallback response matches the primary model's tool-call formatYes, same graders as the primaryNo, no native eval layer
Traces a fallback quality drop to the change that caused itYes, whether a commit reordered the chain or a provider updated a model with no commit at allNo
How it works

How Tessary grades every fallback

One trace model for every provider, one set of graders, and every drop traced to the change that caused it.

  1. Ingest every provider into one trace model

    Every provider’s traffic lands in one trace model built for agents: sessions, turns, traces, and typed observations, all in gen_ai OpenTelemetry conventions.

  2. Land primary and fallback in the same graph

    A primary response and a fallback response from a different vendor become nodes on the same graph, not separate logs compared by hand.

  3. Run the identical graders against both

    Plain-language checks compiled to classifiers, regex checks, tool-error extraction, and any custom grader run the same way regardless of which model produced the trace.

  4. Trace a quality drop to the change that caused it

    Each verdict carries the lineage of the change behind it. Sometimes that is the commit that reordered the fallback chain. Just as often it is a provider updating a model behind the same API name, a change that ships with no commit on your side. Tessary walks the drop back through the graph either way and names the change in the alert.
Questions

Answered, plainly.

Multi-provider agent evals means grading a fallback response with the same graders used on the primary model's response, regardless of which vendor produced it. A routing gateway only checks whether the call succeeded; multi-provider agent evals check whether the response held up, catching format drift, softer refusals, or missing fields that a successful HTTP response wouldn't reveal.
No. OpenRouter's own documentation states its failover reacts to errors, rate limits, and timeouts, not to a response that returns successfully but performs worse. Braintrust's 2026 review of routing platforms found the same gap in Vercel AI Gateway and LiteLLM without a separate eval layer connected. An eval layer closes half of that gap: it tells you the fallback score moved. Tessary closes the rest by telling you which change moved it.
Tessary ingests every provider's traffic into one trace model built for agents, using gen_ai OpenTelemetry conventions, so a primary and fallback response land in the same graph. The identical graders run against both, and when a grader's trend drops, the verdict traces back to the change behind it, whether that was a commit that reordered the chain or a provider-side model update that never touched your repo.
That is the canonical case where nothing in your diff explains the drop. The provider swaps or updates a model behind the same API name, your traces keep flowing, and quality shifts with no deploy on your side. Tessary reads the grader trend on that provider's traffic, and when the trend moves with no matching change in your own lineage, it attributes the shift to the provider-side update rather than sending your team through the repo history.
No, if you already send traces to Langfuse or Braintrust, connect it as an upstream source and pull a slice on demand. They hold the traces and track the scores; Tessary starts where they stop, at saying which change moved a score. Otherwise, the production SDK for TypeScript or Python ingests traffic from any provider directly.
The same graders that run on your primary model's traffic: plain-language checks compiled to classifiers, regex checks, tool-error extraction, and any custom grader you've written. Nothing about a grader changes based on which provider produced the trace being graded.
Any provider whose traffic reaches Tessary, either through direct OTLP ingest, the TypeScript or Python SDK, or an upstream pull from Langfuse or Braintrust. The grading is provider-agnostic by design, since a verdict is a node on the trace graph, not a judgment scoped to one vendor.
Keep exploring

Related pages
Get started

Start grading every fallback response

Connect the provider you fall back to first, with no credit card required. Tessary runs the same graders against it as your primary model, and when fallback quality drops, shows you the change behind it.

Find what broke your agent