Does a model perform best inside its own vendor's harness?
No. A factorial study across 66 model-harness pairings found a model’s own vendor harness is not reliably its best choice, and a pricier harness doesn’t reliably buy a higher score either.
On Terminal-Bench 4, Claude leads GPT by 7.94 points running inside OpenHands but trails it by 30.16 points running inside a harness called PI, and for four of the five models tested, the best harness changed from one benchmark to the next. GPT itself scored higher under PI than under a rival harness called DSH, at under a quarter of the cost per task. One pairing held steady regardless of benchmark: a harness called openJiuwen gave the model Kimi its best score on all three benchmarks tested, by 5.61 to 11.11 points, which the paper traces to how each harness hands a failed step back to the model, since Kimi often returns malformed tool calls and openJiuwen’s scaffold handles that case well.
The lesson isn’t to pick the cheapest or most familiar harness by reputation; model, harness, and task have to be evaluated together, not assumed from any one of them alone.