Commercial LLM routers fail Always-Luna on long open-ended billed prompts—many are constant knobs

On 26 October 2026, Peter Kruger published Autobench’s AutoBench Router Benchmark (ARB) on Hugging Face. The question is not merely whether routing can save money, but a harder one: on long, open-ended prompts billed at real enterprise-like cost, is your “router” choosing a model per prompt—or a knob in front of a constant? The audit locked scoring rules on 29 August 2026, then called routers as shipped between about 30 August and 1 September. The sting is blunt: no commercial setting beats always calling a mid-priced model (Always-Luna) on quality and cost together; several “routers” are effectively constant policies.

What happened

ARB keeps AutoBench’s lookup design: fix 252 open-ended items across ten domains and five difficulty levels; have 20 pool models answer once; score with a ten-model judge committee and store real billed cost. A router only names a model; quality and dollars are read from the matrix—so it cannot improve by writing a better answer. Quality is Delta-Hit (Best Answer rate): whether the stored score lands inside a locked window of the top three in-pool models on that question. Compare against the random line (~64.7% at $0.0115), not against 100%.

The “prize” is the region above random quality and cheaper than random. Always-Luna already sits there (~85.3% at $0.0043)—most of Always-Sol’s quality (87.5% at $0.079) at about 5% of the bill, with no router. An oracle that picks, per prompt, the cheapest model still inside the quality bar reaches 100% at $0.0018; perfect best-model routing is ~100% at $0.034. That gap—about 12 quality points versus Always-Sol and ~98% lower cost—is what routing is sold to collect.

How hard is the evidence

The headline is a negative result, written to be falsifiable: the prize quadrant is empty of commercial routers. None of the seventeen settings beat Always-Luna on both axes; the authors report 10,000 bootstrap resamples with the same empty result. Cheap settings that appear to enter the region sit on the random-quality line because they collapsed onto one flash model; strong settings buy a strong model and cost more than random. The money split matches: cheap routers pile into “cheap miss,” strong ones into “costly hit”; the justifying cell—“cheap hit”—stays a minority.

Behavioural metrics sit beside the single score: five of seventeen settings send ≥90% of prompts to one model (so the quality number is that model’s); Not Diamond’s cost–quality knob is a cliff between knobs 4 and 5 (~80.3% at $0.060 → ~46.6%, below random); OpenRouter Auto and Orca tiers behave more like spend layers than per-prompt routes. Across 59 cross-vendor pairs, median same-pick rate is 0 (exactly 0 for 57.6% of pairs)—if a discoverable right model existed per question, independent routers should converge; high agreement often traces to a shared collapse onto the same cheap model. On the Autobench Router Index at β=0.2, Always-Luna leads; no commercial pass shows positive lift against it.

Steelman and gaps

Steelman: several systems only reach a slice of the 20-model pool (Not Diamond 7, NadirClaw 4, Azure/Brick 3 each), so results describe in-slice behaviour, not a universal ranking; Brick was run as shipped on models it was not tuned for—more distribution shift than best-case. On short multiple-choice boards (RouterArena-style), a cheap model is often already best, so a constant that always names it can look strong. ARB deliberately switches to long open-ended items that cost one to two orders of magnitude more; the commercial story goes quiet. The authors also release questions, the score-and-cost matrix, and decision logs, claiming every table can be recomputed without calling a router again.

Limits remain: 252 prompts, one time window of router versions, an LLM-as-judge committee, and the “top-three window” Delta-Hit definition all shape scores; private enterprise traffic may not match the public pool. The negative result hits these shipped products’ joint promise on this board, not the claim that no future routing algorithm can ever collect the prize. Even the companion white paper title stays interrogative: Do LLM Routers Pay for Themselves?

Who pays, and which questions to ask

For buyers: if a router cannot beat always calling the best mid-priced static policy on your prompts, you bought UX and a billing layer—not a demonstrated per-prompt optimiser. The post lists five diligence questions: what is the static baseline; what is top-model share (especially ≥90%); does the knob have a middle; do two independent routers agree only via shared collapse; and on which board was the claim measured—short exact-match items or long open-ended billed prompts. For cloud and gateway vendors: labelling Auto / quality / low as “intelligent routing” looks, on ARB, more like tiered constants; publishing collapse share and same-pick rates is more honest than another accuracy radar. For the eval community: offline matrices like RouterBench remain useful, but if the eval distribution is so cheap that “always pick cheapest” wins, commercial narratives will be systematically overstated.

What to watch in six months

The authors say the next scientific increment is expensive corpus growth—paraphrase sets, domain and agentic slices—not a longer vendor roster on the same 252 items, because scoring a new router is already a lookup. On the buyer side, falsification is concrete: on your own long-prompt traffic, force a comparison to an Always-“mid-tier” static policy and report quality, real dollars, and top-model share. If you still cannot enter the region above random quality, below random cost, and beat that static policy, ARB is not a scare headline—it is a procurement baseline. The prize is real; as of this board, commercial routers have not collected it.