Research and benchmarks

Mast taxonomy

MAST is a taxonomy of how multi-agent LLM systems fail, from the Berkeley paper "Why Do Multi-Agent LLM Systems Fail?" (arXiv:2503.13657). The authors annotated over 1,000 execution traces from seven open-source multi-agent frameworks and grouped what went wrong into 14 failure modes under three categories: system design issues, inter-agent misalignment, and task verification. Expert annotators agreed on the labels with a Cohen's kappa of 0.88.

Most of the 14 modes live in the interaction between agents, not in any one agent's output. A step gets repeated, a task derails at a handoff, an agent stops before the work is done, a result gets accepted without a check. If you grade each agent's answer on its own, none of that shows up. That's why the paper works from traces rather than outputs, and it's the reason to evaluate a multi-agent system at the trace level.

In practice you use MAST to classify a failed production trace. Walk the trace step by step. At each handoff, check what one agent passed and what the next one actually used. At each stop, check what verified the result. The mode you land on tells you where the fix goes: a design issue means a prompt or role change, a misalignment means changing what agents pass each other, a verification failure means a missing check.

9 questions

Answered, plainly.

What is the MAST taxonomy?The MAST taxonomy sorts 14 recurring ways multi-agent LLM systems fail into three categories: system design issues, inter-agent misalignment, and task verification.answer →Which MAST failure modes show up most in production?Step repetition is the single most common of MAST's 14 modes, in about 16% of the paper's annotated traces, with reasoning-action mismatch close behind.answer →What are the three MAST failure categories?System design issues (41.8% of annotated failures), inter-agent misalignment (36.9%), and task verification (21.3%), each grouping several of MAST's 14 modes.answer →How do I map a production trace to a MAST failure mode?Match the trace to a pipeline stage first, design, interaction, or verification, then to the specific signal inside it; modes with similar symptoms correlate, so one label often isn't enough.answer →Why does grading each agent separately miss multi-agent failures?Because the failure often lives in what one agent's message to another does or doesn't carry, and each agent's own output can look correct when graded alone.answer →Can a deterministic check prevent MAST failure modes without an LLM judge?Mostly not yet. A 2026 attempt at this reported gains everywhere at first, then found over a third of its catches were its own checker's false alarms.answer →Can an AI find which agent caused a multi-agent failure?Barely. The best automated method names the right agent only 53.5% of the time and the exact failing step just 14.2%, on a labeled set of real multi-agent failures.answer →Does MAST's benchmark failure ranking hold up in production?Not closely. A five-month production telemetry study disagrees on which mode leads, premature termination or task incompletion, by metric, with coordination failures nearly absent.answer →Does MAST cover failures between agents that work for different companies?A 2026 study of agent-to-agent systems with separate owners found a 20-mode failure taxonomy; MAST's 14 modes assume one orchestrator and one shared workspace.answer →

Try Tessary on Cloud.

start for free or self-host

free · 10,000 traces a month · 1 GB · 30-day retention · $10 one-time credit for triage and RCA · no credit card