# Swe bench

SWE-bench Pro is a coding test run by Scale AI, released in September 2025. The AI is given
a real bug report or feature request from a real software project, along with the project's
code, and has to write the fix. There are 1,865 tasks from 41 projects. The average fix
touches about 107 lines across four files, so this is closer to a day's ticket than a one-
line patch.

A task counts as solved only if the project's own automated tests pass afterwards: the tests
for the bug now pass, and nothing that worked before is broken. The score is the share of
tasks solved on a single attempt. A 50% score means the AI fixed half the tickets first
time, with no help.

A high score says the model can read an unfamiliar codebase and land a working change. It
doesn't say it will do that on your code. Pro deliberately picks repositories that models
weren't trained on, which is why scores are lower than on the older SWE-bench Verified,
where most top models pass over 70%. About a third of Verified's 500 tasks need only a line
or two, and its tasks are public, so models have seen them.

At launch the best models solved about 23% of Pro. A year later the top of the public
leaderboard is around 60%. A private commercial subset stays harder: the same model that
solves 23% of the public tasks solves 18% of those. That gap is the benchmark's point.

## Questions answered under this concept

- [How much has the top SWE-bench Pro score improved since launch?](https://tessary.ai/answers/swe-bench/how-much-has-the-top-swe-bench-pro-score-improved-since-launch)
- [What is SWE-bench Pro?](https://tessary.ai/answers/swe-bench/what-is-swe-bench-pro)

---

Source: https://tessary.ai/answers/swe-bench
All concepts: https://tessary.ai/answers
From Tessary, agent reliability for AI agents in production: https://tessary.ai