Repeatable success rose, one-time coverage fell

This Microsoft article on Hugging Face is dated October 22, 2026. It follows an earlier post that checked an agent's work against the final database state. This one reports what changed for Qwen3.8-27B on a sealed benchmark after that same kind of executable check was used as a training reward. The result is the judgment: average success rose, tasks completed on all twenty attempts rose, and tasks solved at least once fell. The page states the comparison after 50 GRPO steps. This story does not describe how tasks were selected, how the reward was computed, the trainer path, or the training steps. ThinkingBox-Training is on GitHub. This story does not say how to run it.

ThinkingBox-Bench has 507 tasks. Each is evaluated 20 times, 10,140 attempts in all. pass@1 is average success across those attempts. pass@20 is the share of tasks solved at least once. Observed 20/20 is the share that pass every recorded attempt. For Qwen3.8-27B, average success rose from 51.70 percent to 60.78 percent, and successful attempts from 5,242 to 6,163. Tasks correct on all twenty attempts rose from 38 (7.50 percent) to 107 (21.10 percent), 69 more. Tasks solved at least once fell from 453 (89.35 percent) to 446 (87.97 percent), 7 fewer. The page says repeatable success rose sharply, with a small decline in at-least-once coverage.

The page says no ThinkingBox-Bench task was in the RL training pool. The benchmark was reserved for evaluation. The 507 tasks are an out-of-distribution test: their domains, databases, and most MCP tools were not seen in training. The two sets still share the framework, the simulated user, terminal-state checks, a few tools, and a similar task style. The page says those shared elements limit how far the transfer claim can go. On one corporate booking, a traveler asks to downgrade a room, and company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After fine-tuning, it preserved the booking and payment state in all 20. All expected records matched in 19.

The same page restates a comparison from the first post. It is not a new run of all eighteen models. Kimi-K3 solved 93.89 percent of tasks at least once but completed only 68 of 507 tasks, 13.41 percent, on all twenty recorded attempts. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed the same 241 tasks on all twenty attempts. In this post's own table, GPT-5.6 Sol scored 61.91 percent on pass@1, slightly above the fine-tuned 60.78 percent, but passed 82 tasks on all twenty attempts against 107. Claude Sonnet 4.6 scored 59.19 percent on pass@1 and passed 102 tasks every time. Pass@1 and the all-twenty count rose in all five domains. At-least-once coverage was mixed: it rose in neobank and consulting, and fell in retail, travel, and auto insurance. Auto insurance had the largest drop in that coverage, 7.0 points. Retail had the largest gain on the all-twenty count, 24.5 points.

The dollar figures are a comparative serving index, not an invoice and not a cloud bill. For the same 507 by 20 evaluation, estimated campaign cost fell from 925.80 dollars to 855.58 dollars. Cost per successful attempt fell from 0.177 dollars to 0.139 dollars. Cost per task that passed all twenty attempts fell from 24.36 dollars to 8.00 dollars. The page says serving the fine-tuned checkpoint would be the same as serving the base model. The calculation includes inference cost and excludes the training run and the cost of probing tasks for the pool.