Measuring a 1pp Error With 38.5% of the Effort: The Economics of Cheaper Agent Evaluation in Production

A paper posted to arXiv on September 18 (arXiv:2609.21267) documents a real production dilemma: a production analytics agent serving tens of thousands of monthly active users keeps evolving, but every version update demands a full re-run of the evaluation suite, at rising cost. The authors compared four cost-saving strategies across 574 historical benchmark runs: random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing.

The results come in two layers. First, accuracy: multidimensional 2PL adaptive testing won, executing 200 questions (38.5% of a full run) while holding MAE to 1.03 percentage points. Second, reality: the team deployed not the optimal method but difficulty-stratified fixed subsets, on the grounds of operational simplicity. The fixed subsets also transferred without recalibration to five other agent families and stayed stable across calibration windows as short as one day. The paper closes with practical recommendations for recurring production-agent evaluation.

The value here is pulling evaluation from an academic problem back into an engineering one. Academia asks whether a benchmark measures capability; production asks: every time I change a prompt or swap a model version, how much money and time must I spend to confirm I have not broken the product? Full agent benchmarks can run for hours or days, and a fast-iterating agent product cannot wait — so "38.5% of the work for 1pp of MAE" is a directly seductive number for any team maintaining an agent in production. The methods involved (IRT adaptive testing, historical caching) have been mature in standardized testing for decades; this is the first systematic transplant into production agent evaluation.

The gap between optimal and pragmatic is the most informative part. Theoretically 2PL adaptive testing is better, yet the team shipped fixed subsets. That is not a compromise; it is the real shape of production decisions — the maintenance cost of adaptive testing (question-pool management, calibration updates each round) outweighed its accuracy gain, while fixed subsets are transferable, explainable, and free of calibration infrastructure. For most teams, "good enough and maintainable" beats "optimal but fragile."

State the limits plainly: this is one company's deployment experience with one agent product, and the 574 runs come from its own benchmark. MAE of 1.03pp is within that evaluation's definition; error structures differ across tasks and agent forms, and the transferability claim covers five same-lineage agent families, not all agents. There is also a subtle survivorship issue worth flagging: the 574 historical runs were themselves produced under the team's previous evaluation practice, so the comparisons inherit whatever biases that practice had. The paper itself does not overclaim — it reports the deployment choice, accuracy, and cost separately, which is exactly the honesty evaluation work most needs. For teams building agents today, the takeaway is not which method to copy but the decision frame: measure the cost of a full run, estimate how often you will need it, and choose the cheapest evaluation that keeps your error budget below the threshold your product can tolerate. That frame travels further than any single number in the paper.