Does tool_error count a tool's output as failed just because it mentions the word error?

No. tool_error only counts a call as failed on structural evidence: an error status on the span, a recorded exception, or a result payload that itself declares an error. The word error showing up somewhere in the output text never counts on its own.

That distinction matters because plenty of normal tool output talks about errors without being one: a support tool returning a list of a user’s past error tickets, an agent narrating that it already recovered from a problem, a log line the tool echoes back. Scoring on prose would flag all of that as a failure.

It also catches a case text matching would miss: a tool that catches its own exception, hands back something like {"error": ...} in a clean response, and closes the span without raising anything. That’s a real failure with no exception recorded, and structural evidence is written to catch it anyway.