What is OSWorld?

OSWorld is a benchmark from XLANG Lab at the University of Hong Kong that tests whether an AI agent can finish real work on a computer by reading the screen and driving the mouse, keyboard, and apps. As of October 2026 the current release is OSWorld 2.1, a September 16, 2026 bug-fix update to OSWorld 2.0.

The original OSWorld paper (arXiv:2404.07972, April 2024) had 369 tasks across real web and desktop apps, each checked by its own execution-based evaluation script. The OSWorld 2.0 paper (arXiv:2606.29537, June 2026) moves to 108 long-horizon workflows across seven professional domains, with a median of about 1.6 hours of a person’s time per task, and reports Anthropic’s Claude Opus 4.7 averaging 318 tool calls per task against about 30 on OSWorld 1.0. The tasks run against 31 self-hosted websites, and by the paper’s count 64.8% of them need two or more apps or services. It’s the desktop counterpart of Terminal-Bench, which hands an agent a shell and checks the job afterward.

Since OSWorld 2.0, the task code ships through a gated Hugging Face dataset instead of the public repository, which the maintainers say keeps an agent under test from finding the answers or the grading logic online.