What's the difference between OSWorld and OSWorld-Verified?

OSWorld-Verified is XLANG Lab’s July 2025 fix to the original OSWorld’s grading, the same 369 tasks with roughly 300 reported evaluator bugs corrected rather than the tasks themselves rewritten.

A roughly ten-person team spent about two months on it, and mostly fixed the evaluators rather than the tasks, specifically to keep scores comparable to earlier OSWorld runs instead of starting the count over, the same kind of accuracy question a grader can have independent of how well the agent actually performed. Moving official runs onto AWS, able to run up to 50 environments at once, cut a single evaluation run down to about an hour, per XLANG’s own account.

OSWorld 2.0, which replaced the original OSWorld as the current release in June 2026, is the actual different benchmark: 108 new long-horizon tasks rather than patched old ones, and nothing from XLANG Lab says the two track the same thing, so a score on one says nothing about the other.