# Does BFCL's own test data have grading errors?

Yes. Epoch AI's September 2026 review of BFCL v4 audited a random, stratified sample of 50 of the leaderboard's 5,088 scored questions and found a defect, a case where a fully correct answer could be marked wrong or a clearly wrong one marked right, in 24 of them, 48%.

That's more than double Epoch's own 20% threshold for a "Flawed" verdict, which BFCL received, and the rate wasn't even across categories: from 20% in non-live AST tasks up to 70% in the memory category. A defect in a question doesn't mean every score built on it is wrong, since most of the sample passed review, but it adds real grading noise on top of whatever a model actually got right or wrong, on top of [the scaffold and dataset-age caveats that already apply to any agent benchmark score](/answers/agent-benchmarks/what-is-an-agent-benchmark).

---

Sources:
- Epoch AI benchmark review: Berkeley Function Calling Leaderboard (BFCL) v4: https://epoch.ai/benchmarks/berkeley-function-calling-leaderboard/review (fetched 2026-09-30)

Source: https://tessary.ai/answers/bfcl/does-bfcls-own-test-data-have-grading-errors
More on Bfcl: https://tessary.ai/answers/bfcl
From Tessary, agent reliability for AI agents in production: https://tessary.ai