Does MAST's benchmark failure ranking hold up in production?

Not closely, in the one outside check that exists. MAST’s own ranking comes from benchmark traces: fixed tasks in a harness. A practitioner judged a stratified sample of 376 runs, drawn from five months of a real closed-alpha platform’s production traffic (23,624 runs, 639,381 execution steps), against MAST’s 14 modes, checked against a 39-run hand-labeled gold set: Cohen’s kappa 0.797 on whether a run failed, 71.8% exact match on which mode. Which mode leads depends on which of the study’s own two counts you read: the raw sample puts task incompleteness (1.1) first at 52 runs, ahead of premature termination (3.1) at 38; reweighted to the run population, premature termination leads instead. Either way, the inter-agent coordination failures MAST’s benchmark traces emphasize were nearly absent, partly because the platform’s runs were predominantly single-agent. The judge’s rubric also widens three MAST codes with an added concept apiece: 3.1 also catches hallucination, 3.2 context-window limits, 3.3 cost escalation.

A benchmark score doesn’t transfer to your own agent because the scaffold and task set both change; the same gap shows up here in which failures even occur. The honest read is a case study, not a second benchmark: one platform, one judge.