Should a guardrail result be graded separately from the output?

Yes. A guardrail is answering a different question than the final output is: not “was this response good” but “should this input or output have been blocked,” and folding the two into one grade hides whichever one is actually wrong. An agent can give a correct answer while a guardrail that should have tripped stayed silent, and a passing eval that only scores the final response never catches it.

Build the guardrail’s own eval set the same way you’d build one for a classifier: cases that should trip it and cases that shouldn’t, each labeled with the right verdict, and score precision and recall against that set directly rather than inferring them from downstream behavior. The SDK’s guardrail result carries its own pass or fail independent of what the agent did with it, which is what makes this scoring possible without touching the agent’s output at all. A dataset built from real production traces is a better source for the should-trip cases than ones written from imagination, since a guardrail tuned on hypothetical inputs tends to miss the phrasing real users actually use.