Do I need my own LLM provider key for Tessary to grade traces?

On a self-hosted install, yes. Your org adds at least one model provider’s own key, and that’s what funds the judgment step.

The five classifiers on by default don’t call a model, so your provider choice doesn’t affect them. Two you switch on yourself work differently. Frustration runs on an OpenRouter or TypeSafe key: every message it scores is a call on that key, and if the key is refused or removed it pauses and says so until one works. Groundedness needs no provider key. It scores answers with a public model you run on a Mac with Apple silicon or a GPU instance on AWS, and while that model is down it pauses, then scores what it missed once the model answers again.

Your model-provider spend is the real cost line here. The LLM judgment that reads what the classifiers flag resolves to whichever provider your org holds a credential for, in a fixed priority order, and on a self-hosted install with none configured that step fails closed.