Does BFCL's own test data have grading errors?

Yes. Epoch AI’s September 2026 review of BFCL v4 audited a random, stratified sample of 50 of the leaderboard’s 5,088 scored questions and found a defect, a case where a fully correct answer could be marked wrong or a clearly wrong one marked right, in 24 of them, 48%.

That’s more than double Epoch’s own 20% threshold for a “Flawed” verdict, which BFCL received, and the rate wasn’t even across categories: from 20% in non-live AST tasks up to 70% in the memory category. A defect in a question doesn’t mean every score built on it is wrong, since most of the sample passed review, but it adds real grading noise on top of whatever a model actually got right or wrong, on top of the scaffold and dataset-age caveats that already apply to any agent benchmark score.