This Decoder page is dated Oct 24, 2026. Its published time is 2026-10-24T09:21:39+00:00. Liu and Han at Shandong University held the model fixed and swapped only the harness. The paper is titled "What does a harness buy? Mostly tokens." They write that earlier reports showed gaps of more than 20 points, almost always from a single run. Rerunning the same setup already varies by up to six points on SWE-bench Verified. Zhipu advertises a 2.4-point lead over Claude Code. That 2.4 sits inside the six-point noise.

[1]
A closed charcoal buckle, one muted blue prong
A closed charcoal buckle, blank, with one muted blue prong., AI-generated illustration, not a news photograph

The comparison used five models: Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. The harnesses were the shipped Claude Code, OpenCode, and mini-SWE-agent. Claude Opus 5 was a reference, and only inside Claude Code. On the pool of 447 tasks, the heaviest harness and the lightest sit less than two points apart, and within five points they are statistically equivalent. On the 45 hardest tasks, harnesses differ by two to five solved tasks, while a rerun of the same setup moves the count by up to three. Between two reruns, the median outcome changes on 13 percent of tasks. Between two harnesses it is also 13 percent. Swapping models pushes that to 22 percent.

[1]

Cost is a separate line. With the same model, the same tasks, and the same price list, one task can cost up to three times more. Tool outputs are similar in length. The gap comes from the system prompt and tool descriptions resent at every step: about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. On GLM-5.3-Flash, one Claude Code trial is about 1.81 yuan and one OpenCode trial about 0.56 yuan. On Deepseek-V4-Flash, the light harness takes so many extra steps that the prices meet again. Lower prices for cached tokens shrink the gaps without changing the ranking. The one steady harness effect is OpenCode, trailing by up to nine points on the large pool. About half of that traces to an output limit. The authors have no explanation for the rest. On local models OpenCode also had a tighter time limit, which they flag as uncontrolled.

[1]

One batch was discarded on purpose. In early tests with web access, 71 percent of Deepseek and Qwen attempts pulled the known fix instead of solving the task. The page says those fixes came from GitHub. Claude models almost never did this. The researchers dropped those results, ran the rest offline, and do not claim any maker trained on the test. Thinking was a case where the harness did change the outcome: without it, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. The authors say the grid is too small to explain why. Opus 5 ran in only one harness. A small cross-check on Terminal-Bench 2.1 shows the same pattern.

[1]