# Why do benchmark scores drop when a new version is released?

A new version drops the tasks models have started solving every time and replaces them with harder ones drawn from material the models weren't trained on, so the same benchmark name reports a lower score even though nothing about the model got worse. Public tasks leak into training data over time, which inflates scores on the old set without the model getting any better at the underlying skill.

[SWE-bench Pro](/answers/swe-bench/what-is-swe-bench-pro) is a clean case of it: the older SWE-bench Verified has most top models passing 70%, largely because a third of its 500 tasks need only a line or two and the tasks themselves are public, while Pro deliberately picks unseen repositories and harder tickets, so the best models solved about 23% at launch. BFCL went through the same kind of jump when V4 added agentic and multi-turn sections after earlier versions, which tested one call at a time, had largely saturated. Read a version bump as the benchmark correcting for leakage, not as a real regression in what models can do.

---

Sources:
- SWE-Bench Pro: Raising the Bar for Agentic Coding (Scale AI): https://scale.com/blog/swe-bench-pro (fetched 2026-09-13)
- Berkeley Function Calling Leaderboard (BFCL) V4: https://gorilla.cs.berkeley.edu/leaderboard.html (fetched 2026-09-13)

Source: https://tessary.ai/answers/agent-benchmarks/why-do-benchmark-scores-drop-when-a-new-version-is-released
More on Agent benchmarks: https://tessary.ai/answers/agent-benchmarks
From Tessary, agent reliability for AI agents in production: https://tessary.ai