Does the groundedness classifier only work on agents that use RAG?

No. It also checks summarize and extract calls, which usually have no retrieval step. What every checked answer needs is a source Tessary can read: retrieved documents, or a prompt that holds the document.

A rag_answer call site is checked against the documents retrieved in its trace, or in the nearest earlier trace of the same session, so a follow-up answered from an earlier retrieval still counts. On a summarize or extract call site with no documents, the document is in the prompt, so the answer is checked against the system and user text instead.

What it doesn’t read is tool output, which reaches the model differently from retrieval. A tool result is never treated as a document, so a rag_answer call whose session called a tool and retrieved nothing is skipped: its source is somewhere Tessary can’t read. Emitting tool results as retrieved documents brings those answers into scope.

When an answer does have retrieved documents, a fact that came only from a tool reads as unsupported and can be flagged.