I believe the hardest part of evaluating your agent is turning intent into a judge that reliably measures it. You know how you want your agent to behave, but describing that well enough for another model to judge takes quite a bit of work.
“The agent should respond in our brand’s tone” is a reasonable starting point. But you need to work through real responses to decide what fits, what doesn’t, and where the distinction lies. The same applies to more specific failures, like an agent promising a refund it cannot give. You need to define which promises are unacceptable and give the judge enough context to recognise them.
We’ve added custom classifiers to Tessary to help teams do this. Tessary’s authoring agent works with you using your real traces to understand your intent, turn it into a judge, and refine that judge before it starts monitoring production.
Understanding intent through real traces
We spent a lot of time building the authoring agent around this process. Its primary job is to distill what’s in your head into something an LLM can understand. Your initial description gives it a starting point, and your real traces give both of you examples to work through.
The agent finds traces relevant to your description and guides you through labelling a few. These examples help clarify what the classifier should catch. For a tone classifier, you can look at actual responses and decide whether they fit how you want your agent to speak. The authoring agent uses those decisions to refine the definition.
This matters because some of the distinctions only become clear when you see an example. You might label a response differently from what your initial description suggests, or realise that a rule needs more context. The conversation gives you a way to explain those decisions and carry them into the judge. You don’t have to get the definition right in your first description.
Giving the judge the right context
Once the intent is clear enough, the authoring agent builds the classifier around three decisions: when it should run, what context it needs, and what question the judge should answer.
A classifier looking for inappropriate refund promises needs the relevant answer and the context needed to assess that promise. A tone classifier may need earlier messages to understand whether a response fits the conversation. These checks can involve information across multiple traces in the same session.
The authoring agent handles selecting that context and preparing it for the judge. You don’t have to write the filtering or transformation code yourself. It shows you examples from your traces so you can check whether the classifier is looking at the right parts of the conversation.
Keeping the input focused also keeps the cost low. We want the judge to receive enough context to make the decision without sending the entire session through a large model for every check.
Refining a smaller judge with a teacher model
The classifier needs to be cheap enough to monitor every production session. We use a smaller decision model and a SOTA teacher model during calibration to help refine how it judges your intent.
We use a stronger teacher model to calibrate the decision model against your real traces. You review a few examples to check that the judge matches your intent, without manually labelling a whole dataset.
We test the judge on separate examples that weren’t used during refinement. This helps you assess how well it applies the definition to conversations it hasn’t seen. Agreement with the teacher is useful evidence, though it still depends on how well that teacher understands your intent.
The result is a smaller, cheaper judge built around your product’s specific requirements and tested on its actual conversations.
Monitoring the behaviour in production
Once you create the classifier, it checks the relevant parts of every new session. The filtering happens first, and only the context needed for the decision goes to the judge. This makes it practical to monitor the behaviour continuously at low cost.
You can choose when Tessary should open a finding. By default, it learns the normal flagged rate for each call site and looks for a meaningful change. You can also set a fixed threshold for failures where you already know what rate is unacceptable.
When the classifier detects an issue that meets those conditions, its findings go through the same triage and root-cause analysis as Tessary’s built-in classifiers. Tessary can investigate the affected traces with your connected agent codebase to identify what caused the behaviour.
If the definition needs to change, you can return to the authoring conversation and refine it. Each change creates a new version of the classifier.
Custom classifiers in Tessary
Tessary’s built-in classifiers cover common issues like tool errors, frustrated users, and ungrounded answers. Custom classifiers let you monitor the failures that depend on what your particular agent is meant to do.
The work in the authoring agent is about helping you define those failures well enough for a judge to recognise them. The smaller judge then makes it practical to check for them across every production session.
Custom classifiers are available in Tessary Paid, through self-hosting or managed hosting.