What can retrieval grading over traces not tell me?

It can’t tell you about the chunk that was sitting in the corpus, was genuinely relevant, and never came back at all, because that chunk leaves no trace of its own absence. A trace only records what the retriever actually returned, so grading traces can catch a chunk that’s irrelevant to the question, an empty result set, or a chunk missing an exception clause the question needed.

Measuring true recall, what fraction of the relevant material actually got retrieved, needs something a trace can’t provide: a labeled set of queries mapped to the document IDs that should have come back for each one. Building that set is exactly the slow, hand-labeled work that grading production traces was meant to avoid.

The two methods answer different questions. Trace grading finds retrieval mistakes you can see. Recall measurement finds the mistakes you can’t, at the cost of the labeling effort you were trying to skip.