When do I still need a golden set?

You still need a golden set the moment you want to trust the grader, not just the agent. A hand-labeled dataset is what a grader’s precision and recall are measured against, and that measurement is the only way to know whether the judge you built agrees with a person on what counts as a failure. Production traces can’t supply that, because a trace has no expected answer attached until someone writes one.

Day to day, grading production traces without a golden set works fine for catching failures as they happen. A golden set earns its keep on a narrower job: calibrating the instrument, checking a new grader before it’s trusted, and re-checking an old one after the agent, the prompt, or the judge model changes underneath it. Skip that and you’re trusting a judge that’s never been graded itself.