Agent Reliability

Evals that start without labels

Q: Can I run agent evals without labeled data or a golden set?

Yes. Natural-language-to-classifier signals translate a description of a failure mode into a grader that runs against production traces. For example, "the agent cited a tool result it didn't actually call" becomes a classifier that flags that pattern across all incoming spans. No labeled examples or golden set are needed for the signal to detect failures in live production traffic.

Q: What is the labeling requirement for agent evaluation, and how do I get around it?

Most eval frameworks expect hand-labeled pairs (an input and the correct output) to calibrate what good looks like. The alternative is behavioral signals over production traces: natural-language descriptions of failure modes that Tessary converts to classifiers, which then grade real production spans rather than comparing against a reference output.

Q: What kinds of eval signals work without a labeled dataset?

Tool-error extraction, topic adherence checks, response completeness, and hallucination detection all run without labeled examples. Each is expressed as a natural-language description of the failure mode; Tessary converts it to a per-tenant classifier and grades production traffic. No reference outputs are needed for any of these signal types.

Q: When should I build a golden set for agent evals?

Start building a curated set once you have enough production traffic to sample real edge cases from. Use production spans where a grader already fired as the candidate cases. These are traces the agent handled in a way that warrants review, so they need less annotation effort than cases invented from scratch.

Q: How do natural-language-to-classifier signals work in practice?

You start from a drafted description of a failure pattern: for example, "The agent responded with information not present in the context it had access to." Tessary builds a lightweight per-tenant classifier from that description and runs it against incoming traces. Each trace gets a pass or fail verdict per grader, with no labeled examples needed.

Q: How accurate is a classifier built without labeled examples?

Individual verdicts can be imperfect, and Tessary does not treat any single one as truth. It reads the trend across a grader's verdicts over time: an imperfect grader misfires at a roughly stable rate, so a failure rate that jumps from 3% to 11% flags a real change even when the grader captures intent imperfectly. Traces the grader flags then become the curated set that calibrates it further.

Q: Can agent evals run on existing Langfuse or Braintrust traces without relabeling?

Yes. Connecting Langfuse or Braintrust as an upstream source pulls a chosen slice of existing traces into Tessary, and natural-language signals run against those spans directly. No relabeling, no reformatting, and no export step is needed before the first grader runs. The difference is what happens after: Langfuse and Braintrust track scores over time and tell you a score moved. Tessary's graders feed detection and attribution, so when a trend drops, it walks the drop back to whatever changed.

The labeling requirement is the most common reason eval coverage never starts. Tessary converts plain-language descriptions of failure modes into graders that run against your production traces, no golden set or annotation round required. And when a grader's trend starts falling, Tessary traces the drop back to the change that caused it: a prompt edit, a tool update, or a model provider swap that never touched your repo.

Find what broke your agent See evals that evolve with production

The barrier

The prerequisite that never gets completed

The most common barrier to starting evals is not grading logic or tooling choice: it is having nothing to annotate before the first grader can run. Building a labeled dataset feels like a prerequisite to starting at all, so the prerequisite never gets completed. The result is a team with real production traffic, real failure reports, and no structured way to measure either.

The problem

Labeling blocks coverage

Most teams already operate without benchmark sets, yet standard eval frameworks assume reference outputs exist before the first grader runs. Under a label-first approach, three things must arrive before any coverage exists: an observation pipeline, an annotation schema, and agreement on correctness criteria.

75%

of production agent teams interviewed in a 2025 study (15 of 20) evaluate without benchmark sets, relying on A/B tests, user feedback, and production monitoring

Measuring Agents in Production, arXiv 2512.04123

<10%

of organizations have scaled agentic AI in any individual function

Adaline, 2026 evaluation guide

Two approaches

Label-first vs signal-first

Both approaches produce graders. The difference is when coverage starts relative to when you have data.

Label-first approach

Coverage waits for the golden set

Cannot grade any trace until reference outputs exist
Annotation schema must be agreed upon before the first grader runs
Golden set construction is a prerequisite, not a deliverable
Coverage begins when the dataset is complete, not when production traffic starts

Tessary

Signal-first approach

Coverage starts at first trace ingest

Natural-language failure description becomes a per-tenant classifier immediately
Grades production traces without any reference outputs
Coverage begins from the first trace ingest, before annotation
Golden set can be built incrementally from flagged production edge cases

How it works

Signal-first coverage

Tessary drafts failure descriptions from your traces, compiles them to classifiers, and watches the resulting trends. No reference outputs, no annotation round, no golden set required to start.

Connect your trace source
Send production traces via OTLP or pull from Langfuse or Braintrust. No re-instrumentation required if your agent is already sending traces. Every incoming span is available for grading from the first ingest.
Correct a drafted failure description
Tessary reads a sample of your traces and drafts plain-language descriptions of the failure modes it sees: "The agent cited a tool result it did not actually call," or "The response did not address the user's question." You correct the drafts rather than write them from scratch. These descriptions are your intent for the agent, stored outside its code, and they sharpen as bugs are fixed and edge cases land.
Run the classifier
Tessary converts each description to a per-tenant classifier and runs it against incoming traces. Each trace gets a pass or fail verdict per grader. No labeled examples are needed for the classifier to run.
Build a curated set from production edge cases
Once you have enough traffic, sample traces where a grader fired as candidates for a curated dataset. These are cases the agent handled in a way that warrants review, so they need less annotation effort than cases invented from scratch.
Trace a falling trend to its cause
The graders are the sensor, not the destination. Tessary watches each one's trend over time, and when a trend drops, it traces that quality dimension back through the agent's chain to the change behind it: a prompt edit, a dependency bump, an upstream agent's shift, or a tool that started returning different data.

Signal types

Four signals, no labeled data

Each is expressed as a natural-language description of a failure mode. Tessary builds a per-tenant classifier from it and grades production traffic against real trace structure.

Tool-error extraction

Identifies cases where a tool call returned an error the agent ignored or misrepresented. The classifier grades against the tool call result and the agent's subsequent response. No reference output needed.

Topic adherence

Flags traces where the agent's response fell outside the expected domain. Expressed as a description of the scope boundary, the classifier checks each response against that constraint.

Response completeness

Detects traces where the agent addressed only part of the user's question. The classifier checks the response against the original query structure, not against a reference answer.

Hallucination detection

Catches cases where the agent cited facts not present in the context it had access to. The classifier grades against the trace's own span data: the context and the response the agent produced.

FAQ

Questions about label-free evals

Yes. Natural-language-to-classifier signals translate a description of a failure mode into a grader that runs against production traces. For example, "the agent cited a tool result it didn't actually call" becomes a classifier that flags that pattern across all incoming spans. No labeled examples or golden set are needed for the signal to detect failures in live production traffic.

Most eval frameworks expect hand-labeled pairs (an input and the correct output) to calibrate what good looks like. The alternative is behavioral signals over production traces: natural-language descriptions of failure modes that Tessary converts to classifiers, which then grade real production spans rather than comparing against a reference output.

Tool-error extraction, topic adherence checks, response completeness, and hallucination detection all run without labeled examples. Each is expressed as a natural-language description of the failure mode; Tessary converts it to a per-tenant classifier and grades production traffic. No reference outputs are needed for any of these signal types.

Start building a curated set once you have enough production traffic to sample real edge cases from. Use production spans where a grader already fired as the candidate cases. These are traces the agent handled in a way that warrants review, so they need less annotation effort than cases invented from scratch.

You start from a drafted description of a failure pattern: for example, "The agent responded with information not present in the context it had access to." Tessary builds a lightweight per-tenant classifier from that description and runs it against incoming traces. Each trace gets a pass or fail verdict per grader, with no labeled examples needed.

Individual verdicts can be imperfect, and Tessary does not treat any single one as truth. It reads the trend across a grader's verdicts over time: an imperfect grader misfires at a roughly stable rate, so a failure rate that jumps from 3% to 11% flags a real change even when the grader captures intent imperfectly. Traces the grader flags then become the curated set that calibrates it further.

Yes. Connecting Langfuse or Braintrust as an upstream source pulls a chosen slice of existing traces into Tessary, and natural-language signals run against those spans directly. No relabeling, no reformatting, and no export step is needed before the first grader runs. The difference is what happens after: Langfuse and Braintrust track scores over time and tell you a score moved. Tessary's graders feed detection and attribution, so when a trend drops, it walks the drop back to whatever changed.

Keep exploring

Get started

Start evals before you have labeled data.

Connect your trace source and Tessary drafts the first graders from your production traffic. When one of them starts trending down, you learn what changed, not just that a score moved. The first graders run on real spans before you provide a single labeled example or credit card.

Find what broke your agentAlready on Langfuse or Braintrust? Pull existing traces without relabeling. When a cause hunt hits a gap in the traces, that gap is the next node to instrument.