Swe bench

SWE-bench Pro is a coding test run by Scale AI, released in September 2025. The AI is given a real bug report or feature request from a real software project, along with the project's code, and has to write the fix. There are 1,865 tasks from 41 projects. The average fix touches about 107 lines across four files, so this is closer to a day's ticket than a one- line patch.

A task counts as solved only if the project's own automated tests pass afterwards: the tests for the bug now pass, and nothing that worked before is broken. The score is the share of tasks solved on a single attempt. A 50% score means the AI fixed half the tickets first time, with no help.

A high score says the model can read an unfamiliar codebase and land a working change. It doesn't say it will do that on your code. Pro deliberately picks repositories that models weren't trained on, which is why scores are lower than on the older SWE-bench Verified, where most top models pass over 70%. About a third of Verified's 500 tasks need only a line or two, and its tasks are public, so models have seen them.

At launch the best models solved about 23% of Pro. A year later the top of the public leaderboard is around 60%. A private commercial subset stays harder: the same model that solves 23% of the public tasks solves 18% of those. That gap is the benchmark's point.