How many traces do you need before you're likely to catch a rare failure once?

With a 2% failure rate and a 5% review sample, the numbers behind the concept’s own example, you need about 3,000 agent sessions before you’re 95% likely to have caught the failure once. The formula is N = ln(1-c) / ln(1-p·s): c is the confidence you want, p is the failure rate, and s is the share of sessions you actually review. Here p·s works out to 0.001, one session in a thousand both fails and gets looked at, so ln(0.05) / ln(0.999) comes to roughly 2,995 sessions.

Push either number down and the wait gets much longer. Halve the review sample to 2.5% and you need about twice as many sessions for the same confidence. A rate this low doesn’t announce itself in a small sample. It just takes volume, and volume takes time you may not have before a failure like this compounds.