How do I pick the confidence threshold for letting a decision run automatically?

There’s no single number for a whole system. TypeSafe’s own guidance is to gate different actions at different levels depending on what a wrong call costs there, a mistake on an FAQ answer and a mistake on a refund don’t deserve the same bar, and to set each threshold by plotting confidence against accuracy on your own labeled data rather than picking a round number like 0.8 in advance.

The pattern is simple once a threshold exists: send the confident cases straight through and route the rest to a person or a more expensive reasoning model, something like if answer.confidence < 0.8: route_to_human_review(ticket). Checking whether that confidence is calibrated in the first place has to happen before any threshold means anything, since a 0.8 that’s actually right 60% of the time isn’t a safe cutoff wherever you set it.

Start conservative and adjust as you observe real results. Tessary’s own classifiers escalate the same way: crossing a threshold writes a finding for triage rather than acting on its own, because the cost of a wrong automatic call is higher than the cost of a short delay.