What is SWE-bench Pro?

SWE-bench Pro gives an AI a real bug report or feature request plus the project’s own code, and counts the task solved only if the project’s automated tests pass afterward, with nothing that worked before broken, on a single attempt with no help. Scale AI built it from 1,865 tasks across 41 repositories chosen specifically because models weren’t trained on them.

Unlike earlier coding benchmarks, which filtered out ambiguous or underspecified bug reports to keep tasks clean, SWE-bench Pro keeps them: a human expert clarifies the report enough to be solvable while leaving the original technical challenge intact, closer to a real ticket than a curated one. Its language mix is uneven too; Python and Go tasks see meaningfully higher success rates than JavaScript and TypeScript ones for the same models. What an agent benchmark score actually tells you covers why that score still doesn’t transfer to your own codebase.