How do you measure agent reliability in production?

You measure it as a rate, not a single verdict. Because agents are nondeterministic, one run tells you little: the same input can produce a good answer one time and a bad one the next. What matters is the distribution, so the practical measure is something like the share of production runs that meet the bar you set for correct behavior, tracked over a rolling window so you can see it move. That requires judging the content of each run, not just whether it completed, since a wrong answer and a right one both return a normal status. It also requires a consistent baseline: the same measure applied before and after a change is what lets you tell a real shift in reliability from ordinary run-to-run noise. A single spot check, on one run or one day, cannot do either job.