Osworld

OSWorld is XLANG Lab's benchmark for AI agents that use a computer the way a person does: reading the screen and working the mouse, keyboard, and apps on a real virtual machine. The original 2024 version had short tasks that frontier agents now mostly pass. OSWorld 2.0, released in June 2026, replaced them with 108 long workflows that take a person a median of about 1.6 hours, and OSWorld 2.1, a September 2026 bug-fix release, is the current version.

Each task is graded against a list of checkpoints, so a result is reported either as the share of tasks fully completed or as partial credit, and the two can sit far apart. As of October 2026 no agent fully completes half of the tasks.

Its strength is realism: tasks span several apps and websites, some change while the agent works, and the task code is kept out of the public repository so agents can't look up the answers. Its weakness is cost and setup: runs take hundreds of steps, teams self-host the task websites, and scores from different releases, task subsets, or agent harnesses don't compare.