Why are FrontierCode scores lower than SWE-bench Pro scores?

FrontierCode’s top score, 54.6%, sits below SWE-bench Pro’s 61.5% because a passing SWE-bench Pro patch only has to make the project’s own tests pass, while FrontierCode adds scope, a check on whether the patch touches only what it needs to, as a second merge-blocking requirement: fail either one and the attempt scores zero. Style and convention are graded too, but as weighted quality signals that lower a passing score rather than as blockers that can zero it.

Cognition built FrontierCode specifically because that gap exists. Its launch post names SWE-bench Verified and SWE-bench Pro as “the first generation of coding benchmarks” and says they “only test functional correctness, not quality,” pointing to METR’s finding that high scorers on those benchmarks “often produce patches that wouldn’t be accepted by human maintainers.” FrontierCode’s scope blocker is built to catch exactly that: a correct fix that touches files it shouldn’t.

The two numbers aren’t measuring the same bar, so a lower FrontierCode score isn’t models getting worse at coding between the two benchmarks. Changing only the harness is enough to move a model’s score on one identical benchmark; a stricter grading bar across two different benchmarks does more of the same.