# How do you find a failure mode nobody defined a check for?

By clustering traces on how the failure behaves instead of grading them against a named check, since a check only ever catches a failure someone already named and wrote a rule for. Group enough traces by the shape of what went wrong rather than by topic, and a pattern nobody defined shows up as its own cluster before anyone names it.

This inverts the usual order. A grader starts from a named failure mode and looks for it in each trace; clustering starts from the traces and lets a recurring failure surface on its own, named or not. A 2025 framework built around this, AgentCompass, ran cross-trace clustering over agent execution logs and surfaced failure patterns that the TRAIL benchmark's own human annotations had missed, evidence that some of what an agent does wrong only becomes visible across traces, not inside one.

A cluster isn't a finding by itself. Someone still has to look at what landed together and decide it's a real problem before it becomes [a grader](/answers/graders/what-is-a-grader) that catches it going forward. And clustering only finds what recurs; a genuine one-off stays invisible either way.

---

Sources:
- AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production (arXiv:2509.14647): https://arxiv.org/abs/2509.14647 (fetched 2026-09-10)

Source: https://tessary.ai/answers/failure-modes/how-do-you-find-a-failure-mode-nobody-defined-a-check-for
More on Failure modes: https://tessary.ai/answers/failure-modes
From Tessary, agent reliability for AI agents in production: https://tessary.ai