Hugging Face thumbnail for Harness Adapters, showing logos for Hermes, pi, OpenCode, and Qwen3.8-27B on a purple background.
Thumbnail from the Hugging Face harness adapters post., Hugging Face

On 20 November 2026, Hugging Face published a recipe for what it calls harness adapters: small LoRA add-ons trained with reinforcement learning while the model is running inside the actual agent software it will use later. A harness, in this post, is that software loop, such as pi, Hermes Agent, or OpenCode, not a metaphor. On Qwen3.8-27B, with thinking turned off, the company's own specialist adapter for the pi loop raised pass@1 on the 51 hardest SWE-bench Pro tasks from 39.2% to 64.7%. It also cut tool calls sharply on tasks both models already solved.

The implied lesson is easy to over-read. This is not evidence that Qwen3.8-27B is now a better model in general. It is evidence that training inside one tool loop, on a filtered set of coding tasks, can raise accuracy and cut wasted steps in that same loop. A merged adapter that tries to serve all three loops keeps some of the gain and loses a large part of it under pi. A separate test on AutomationBench, which uses its own agent loop, stayed close to the base model.

[1]

Hugging Face trained one LoRA adapter per loop, on Qwen3.8-27B, using TRL's AsyncGRPOTrainer. The three loops are pi 1.5.2, OpenCode 2.2.7, and Hermes Agent v0.22.6. Thinking mode was off during training. The reward was the verifier's pass or fail, plus a small 0.1 bonus for using fewer tools. Each adapter run used two NVIDIA B200 GPUs. Across the three runs, training totaled 598 steps, 78,114 samples, and 167 million tokens. A safety filter called safeagents checked 1.3 million tool calls and blocked 647 of them.

The training tasks were built to avoid a common trap. Of 442 scored tasks, the base model solved 151 every time and 120 never. Only the 171 it sometimes solved were treated as qualified, because a method that scores attempts against each other learns little when every attempt is a pass or every attempt is a fail. Hugging Face also dropped broken verifiers. The final mixture was 750 tasks from its own environments and 300 from Termgrade.

The evaluation is run inside the same Harbor and OpenEnv stack, one attempt per task, at temperature 0.8, with thinking off unless stated. The team is explicit that scores should not be compared across loops. Pi does not keep previous reasoning turns, while OpenCode and Hermes do, which mainly affects SWE-bench Pro when context fills up.

On the in-loop tests, all three specialists beat the base model. On SWE-bench Pro's hard 51, hermes went from 29.4% to 43.1%, and opencode from 35.3% to 51.0%. On all 89 Terminal-Bench 2.1 tasks, pi gained 16.3 points and hermes 14.1. Opencode's overall gain was 6.0 points, and 13.6 points on the 66 tasks that were not used to pick its checkpoint. On tasks both the adapter and the base model solved, tool calls fell 56 to 72 percent on SWE-bench Pro and 53 to 70 percent on Terminal-Bench 2.1.

[1]

Hugging Face also merged the three LoRAs into one super-adapter of rank 96, using half-strength updates. That half weight was chosen on Terminal-Bench 2.1 with earlier checkpoints, so the Terminal-Bench merge numbers are, by the authors' own note, slightly optimistic. SWE-bench Pro was not used to pick the weight.

The merge comparison, on the tasks every run graded, shows the tradeoff clearly. With thinking off, the pi specialist reaches 64.0% on SWE-bench Pro hard, while the super-adapter falls to 46.0%, only a little above the 40.0% base model in that table. Hermes and OpenCode keep more of their gains, and in one Hermes cell the merge even edges the specialist. Hugging Face's advice is direct. If you use one loop and keep thinking off, take that loop's specialist.

With thinking on, at the default xhigh effort, the super-adapter, which was not trained in thinking mode, still beats or ties the thinking base model on the six loop-and-benchmark pairs, with a mean pass@1 of 55.5% to 63.5%. Under pi on SWE-bench Pro, thinking alone already reaches 74%, and the super-adapter ties. Efficiency remains. With thinking off, the super-adapter makes 53 to 73 percent fewer tool calls on tasks both it and the base model solve.

When the same adapters were run on 300 AutomationBench business-automation tasks, in AutomationBench's own loop, the picture flattened. The base model scored 23.0% pass@1. The pi and OpenCode adapters were slightly ahead, at 27.0% and 25.3%. The merged adapter was almost level, at 23.7%. Hermes was behind, at 17.7%. Hugging Face treats most of these gaps as noise, except perhaps Hermes, and concludes that training inside the coding loops neither clearly transfers to this setting nor clearly damages it.

[1]

The authors also inspected trajectories and asked what the model actually learned. In non-thinking mode, the adapters often write a short stretch of text before a tool call, sometimes in the manner of a reasoning trace, while the base model tends to call tools with almost no preface. They describe this as a kind of reasoning leakage, the non-thinking model finding a way to behave a little like its thinking version. They do not treat this as invalidating the scores, because the comparisons use a fixed token budget. They do say a follow-up should train thinking and hybrid models directly.

Two tables in the post also use slightly different task subsets and therefore slightly different headline percentages, such as 39.2% to 64.7% versus 40.0% to 64.0% for the pi specialist on SWE-bench Pro hard. The difference is small, but it is a reason to cite the table you mean.

[1]

The leakage note is the right counterargument. If a large part of the gain is cheap reasoning under a thinking-off label, then the headline gap between a specialist adapter and a thinking base model may shrink once thinking is trained on purpose. The pi SWE-bench Pro tie at 74% already hints at that ceiling in one cell. The AutomationBench result hints at another limit. Change the work and the loop, and the adapter is no longer a general upgrade.

The narrower claim still stands. Inside the same agent software and the same family of coding tasks, a small adapter trained in that loop can raise pass@1 and cut tool calls. Hugging Face has posted adapters, training tasks, and evaluation trajectories. Those are the objects to check.

[1]

The useful test over the next six months is whether other teams can reproduce the in-loop gains on their own software, and whether adapters trained with thinking on still cut tool calls without the leakage workaround. Watch also what happens when the target loop itself changes version. That would show whether the recipe is tied to one snapshot of pi, Hermes, or OpenCode, or whether it survives an ordinary software update.

[1]