Coding Agents Have Converged: A New Audit Finds SWE-bench Can No Longer Rank Its Leaders

A paper posted September 16 argues that the top of the SWE-bench Verified leaderboard can no longer be ordered. "Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead," by Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo and Siyuan Guo, audited 254 public submissions across four splits without running a single model. It was accepted at ADMA 2026's special session on responsible data intelligence and has been circulating on Hacker News this week.

The core finding is a convergence result. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving only 164 instances that distinguish their outcomes; the frontier solution sets have a median nesting coefficient of 0.935 against a score-implied baseline of 0.774. Under exact paired McNemar tests, none of the 29 adjacent pairs among the Verified top thirty separate at alpha = 0.05 — the smallest adjacent p-value is 0.545. The larger Test split does separate 14 of 23 adjacent pairs, which matters for interpretation.

Scores also depend on the evaluated model-scaffold pair. Reconstructing the design from public metadata, the authors find observed within-model scaffold ranges of up to 29.8 percentage points — larger than the 8.8-point spread across the entire top thirty — and scaffold orderings that reverse depending on the model. Their descriptive tiering yields three groups, or two after Holm correction, and they stress that non-rejection does not establish equivalence. Internal consistency tells the same story from another angle: KR-20 is 0.994 across the full pool but collapses to 0.475 across the top ten. The instrument can rank the field; it can no longer rank the leaders. The paper also prices doing better: separating adjacent pairs with 80% power in a paired design would require roughly 734,000 instances — a figure the authors carefully describe as an independent-sample power reference, not a paired-test requirement. That number is the quietest and most damning sentence in the paper.

The paper's remedy is a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity and prices new instances, plus a recommendation to report comparison-set-specific resolution and model-scaffold provenance instead of reading small aggregate gaps as established rank differences. The authors release the full pipeline, including the per-instance verdict partition, so any lab or procurement team can re-run the audit on its own comparison set rather than trusting a single aggregate number.

This is a benchmark-saturation paper, not a capability paper, which is exactly why it is uncomfortable for the industry. SWE-bench drove the coding-agent race for two years; vendors and procurement teams quoted it as if adjacent ranks were real. The authors' contribution is to turn an intuition into a reproducible statistical fact: when the best systems solve the same problems and fail the same ones, their ranks mostly encode noise. The counter-example in their own data — Test split still separates 14 adjacent pairs — suggests the problem is not SWE-bench itself, but that resolution at the frontier has run out. The natural next move is not a better leaderboard. It is a harder one.