Why doesn't monitoring catch a silent agent failure?

Standard monitoring watches whether a run completed, not what it said: error rate, latency, and uptime all read healthy on a run that returned the wrong answer. None of those three signals reads the content of the response, so a run that returns the wrong answer in a normal amount of time with a 200 status looks identical to one that got it right. An agent that takes a shortcut, quotes a stale number, or claims it finished a task it didn’t touch still closes its trace clean, because closing clean only means the transport layer didn’t break.

Catching it needs a different kind of check, one that grades what the agent actually said or did against what it was supposed to do, not whether the request round-tripped. That means reading the output itself, not its status code, and doing it on enough of production that a failure showing up in a small share of runs has a real chance of getting seen.

keep reading

More on this.

Send us the traces you already emit.