How do you detect silent failures in production?

You grade the content of the run against what it was supposed to accomplish, not the status the run returned. That means a check that reads the final answer, or the tool calls and their actual effects, and compares them to the task, separate from whatever the trace’s own spans report about themselves. A trace that closes clean tells you the transport worked; it says nothing about whether the agent did the right thing.

The second requirement is coverage. Silent failures are rare by definition, so a check that only runs on a small sample of production traffic can go a long time without landing on the session that failed. The check itself also has to be cheap enough to run broadly, because a slow, expensive grader on a thin slice of traffic has the same blind spot as no grader at all: it just misses quietly instead of loudly.