AA-Briefcase time per task: falsify the Elo–latency Pareto, not “faster = stronger agent”

On 24 Jun 2026 Artificial Analysis published a time-per-task reading of AA-Briefcase. The benchmark itself scores deliverables inside multi-week knowledge-work projects — financial models, board decks, design mock-ups — written by industry experts.

The falsifiable reading is narrow. If, under the published method, score leaders are not systematically slower, or tool execution is far above the stated ~12% time share, the product implication “high score ≈ expensive time” needs a rewrite. This piece does not treat “agents already finish multi-week projects for humans” as proven.

Hard facts (AA time-per-task post, 24 Jun 2026, cross-checked with the 18 Jun 2026 launch post):

• Metric: average time per task = evaluation token usage ÷ representative output speed + tool execution time recorded in eval.
• Claude Opus 4.8: among the highest-scoring available models then, ~23 minutes per task on average.
• Several GPT-5.5 reasoning settings sit on the Elo–time Pareto; xhigh in particular takes ~11 minutes per task, about half of Opus 4.8, while ranking top-five on overall Elo (GPT-5.5 xhigh Elo 1159 in the post).
• GLM-5.2: Elo 1261, ahead of that GPT-5.5 xhigh, at ~16.3 minutes; also the top open-weights model, with MiniMax-M3 next at Elo 1113.
• If Claude Fable 5 were still available: authors estimate ~28.5 minutes per task (while live: ~91 output tok/s, ~3.1 minutes tool time per task, ~139k output tokens per task).
• Structure: tool calls and execution are only ~12% of total time; the rest is explained by output verbosity, turns, and inference speed.
• Benchmark shape (launch post): four private multi-week scenarios, 91 tasks; a public fifth scenario is demonstrative only and does not count toward official scores.

Evidence strength: designer-run, designer-reported. Time is estimated wall-clock from tokens over nominal speed plus tool time, not end-user latency after real cloud queues. Elo aggregates rubric pass rate with pairwise analytical and presentation judgments.

Mechanically, the post pins most delay on the model’s own verbosity, turn count, and inference speed, not on the tool sandbox. That has a direct buying implication: what you often purchase is “a high score that is willing to think longer,” not “tools that eat most of the clock.”

Steelman: if the Pareto chart is stable, buyers can trade explicitly between Opus-class high-and-slow tiers and options such as GPT-5.5 xhigh or GLM-5.2 that are “good enough” on score with better time or open weights, instead of reading only the leaderboard name.

Strongest steelman: putting time on the same plane as Elo is closer to real deployment cost than score-only tables; open-weights GLM-5.2 appearing near both high score and the Pareto weakens “only closed models can do long-horizon knowledge work.”

Steelman risks: tasks run independently without carrying the model’s own prior deliverables, unlike human multi-week state. Private items cut contamination and raise external reproduction cost. If “representative output speed” diverges from a given cloud’s real throughput, wall-clock shifts systematically. Fable 5 estimates lean on historical speed once the model is gone. Figures here are a June 2026 snapshot; the live leaderboard will move.

For procurement and platform teams, the six-month check is whether high-scoring models still land near the 10–30 minute per-task band on internal workflows, and how much score falls when verbosity and turns are cut. For eval readers, read Elo, latency, and cost together instead of screenshotting the top name. For model builders, if tool time stays a small share versus output, the optimization lever is likelier “when to stop and how short to write” than adding more tool types.

The time-per-task note splits “agents are strong” into a checkable score–time tradeoff. The falsifiable focus should be: whether high scorers stay systematically slower, whether tools really take only about a tenth of the clock, and whether open-weights models still sit near the Pareto. If those hold on updated boards, “faster = stronger” fails; what survives is an explicit Elo–latency trade.

【No.2】