Why do benchmark scores drop when a new version is released?

A new version drops the tasks models have started solving every time and replaces them with harder ones drawn from material the models weren’t trained on, so the same benchmark name reports a lower score even though nothing about the model got worse. Public tasks leak into training data over time, which inflates scores on the old set without the model getting any better at the underlying skill.

SWE-bench Pro is a clean case of it: the older SWE-bench Verified has most top models passing 70%, largely because a third of its 500 tasks need only a line or two and the tasks themselves are public, while Pro deliberately picks unseen repositories and harder tickets, so the best models solved about 23% at launch. BFCL went through the same kind of jump when V4 added agentic and multi-turn sections after earlier versions, which tested one call at a time, had largely saturated. Read a version bump as the benchmark correcting for leakage, not as a real regression in what models can do.