Can an AI find which agent caused a multi-agent failure?

Barely. On Who&When, a labeled set of 184 real multi-agent failures, the best automated attribution method named the responsible agent correctly only 53.5% of the time, and the exact failing step just 14.2% of the time. Frontier reasoning models tested against it, including OpenAI’s o1 and DeepSeek’s R1, did no better: the paper’s own verdict is that none of the methods it tried reach practical usability.

The paper’s own explanation for the task’s difficulty is domain expertise and scale: mapping an evaluation result back to the responsible agent and step takes specialized judgment, and the number of components a reviewer has to check only grows as the system does. Finding which change caused a quality drop in a single agent already means separating a real regression from everything else that could explain it; a multi-agent failure adds a second unknown, which agent to blame, on top of that search.

Until automated attribution improves, walking the trace by hand, checking what each handoff passed and what the next agent actually used, still beats trusting a tool’s verdict.