要点
- Hugging Face blog (2077AI · UC Berkeley RDI, 3 Nov 2026): CyberArena runs live symmetric attack-and-defense — two agents get identical vulnerable services, three 20-minute rounds of simultaneous attack and patch.
- Campaign: 466 valid matches among 12 frontier agents on 40 ranked challenges (of 760 submitted; through batch 22).
- GPT-5.6 Sol expected score 78.6 [76.8, 80.2] vs Claude Opus 5 at 73.8; keep rates 83%/81% vs 43–61% for the rest; median first restart 2.3 min vs 3.9–12.3.
- Audit error modes: unverified fix claim 23% of audited matches; partial patch 21%. Thesis fails if static attack-only scores still rank the field as well.
On 3 November 2026, Bangya Liu, Yueyang Pan and Hanhong Zhao (2077AI · UC Berkeley RDI) published CyberArena on Hugging Face: a static cut of the cyber-arena.ai report that replaces the jeopardy CTF exam with a contest. Two agents receive identical copies of a vulnerable service; over three rounds each attacks the other's copy while patching its own. The arena — not the agents — verifies every capture and every health check.
The falsifiable claim is narrow. Live symmetric combat separates frontier agents where solo flag-capture benches saturate, and the ranking edge is defense speed and quality — flag retention and time-to-first-restart — not solo exploit capture. If static attack-only scores keep predicting CyberArena standings equally well, that claim fails.
[1]What the arena actually scores
Most cybersecurity agent benches are exams: one agent, one fixed target, one role. CyberArena's match plants a fresh objective each round, opens an attack network between twin services, and runs attack and defense at the same time for 20 minutes × 3 rounds. Patches persist across objective rotation; last round's flags are worthless. Availability is scored by an external health poller — process, TCP, HTTP, plus challenge-specific workflow checks — so an over-blocking "fix" that breaks the SLA costs points (the paper finds over-blocking in 8% of audited matches).
Playing every pairing on every challenge is infeasible (one match occupies two agents for an hour). The campaign therefore selects matches by expected information gain after pilot batches, then fits a Bradley–Terry-plus-item-response model whose only round-level observations are capture Y and health-pass rate Z. An entrant's standing is expected game score against every other entrant and every reference challenge under the posterior — which is how two agents that never met can be ranked.
[1]What the 466 matches showed
Of 760 submitted matches, 466 were valid through batch 22 across 12 agents and 40 ranked challenges. GPT-5.6 Sol (OpenAI Codex harness) leads at 78.6 [76.8, 80.2], five points ahead of Claude Opus 5 (Claude Code) at 73.8, and held first place at every one of the 22 cumulative refits. A plain Bradley–Terry on wins/draws/losses agrees closely (Kendall τ = 0.85) but swaps the top two — because it counts only who won, while expected score also credits margins; the top-pair inversion is not significant (p = 0.41).
Attack and defense abilities correlate (r = 0.85), yet the leader's edge is defense. GPT-5.6 Sol kept its flag in 83% of rounds and Claude Opus 5 in 81%, against 43–61% for the others. Capture rates at the top are almost tied (70% vs 69%); the spread opens on keep rate and on speed: median time to first completed service restart in round 1 is 2.3 minutes for Sol versus 3.9–12.3 for the rest. In one audited rceaas replay, Sol captured at 0.7 min and patched at 1.2 min while Claude Opus 4.8's first patch arrived at 18.9 min — after the flag was gone.
Behaviorally, defenders do respond: when a defender lost its flag in rounds 1–2, it inspected incoming attacks in 67% of cases and restarted before the next round in 57%; those that restarted kept the flag in 60% of following rounds versus 9% that did not (association, not causal proof). The audit's most frequent errors are the unverified fix claim (23% of audited matches — reporting that a patch passed checks while reading pre-restart logs) and the partial patch (21% — closing one entry point while leaving another observed route open).
[1]Steelman and gaps
Steelman. Jeopardy CTFs are cheap, reproducible, and comparable across labs. If the field still sorts almost the same way on Cybench-style solo capture as on CyberArena expected score, then the expensive live arena is measuring the same latent ability with more noise — and the "defense is the difference" story is a reframing of correlated attack skill, not a new axis. A plain win/loss BT already nearly recovers the order; that is evidence the contest is not orthogonal to simpler rankings.
That objection has force, and the authors partly concede the correlation. What it does not erase: keep-rate and restart-time gaps that solo attack scores do not directly observe; adversarial patching that kills memorised exploits; and health-checked repairs that punish over-blocking. The thesis therefore does not claim "attack is irrelevant" — it claims that conditional on similar capture rates, defense speed/quality is what separates the top of the live table, and that static attack-only benches will miss that residual once they saturate.
Limits the blog itself lists: only 12 fixed model–harness pairs; pairwise single-service matches (no volumetric DoS, no multi-agent swarms); an automated audit of 2,397 findings that is mechanically checked but not fully human-confirmed; and a snapshot through batch 22 while the live leaderboard keeps moving. Harness and model are entangled — both Gemini Flash entrants "slept" through active rounds in about a third of matches and share a harness.
[1]Who should care, and what to watch
Buyers of "autonomous security agent" demos should treat saturated jeopardy scores as a necessary but insufficient screen. Procurement that only asks "can it capture the flag on a frozen box?" will over-weight exploit recall and under-weight patch latency, SLA-preserving repairs, and response to a live opponent. Red/blue team tooling vendors that ship agent loops should expect customers to ask for keep-rate and time-to-first-valid-restart, not just capture rate.
Watch next: whether attack-only ranks continue to predict CyberArena standings as the live board moves; whether Recursive Adversarial Improvement — training on the arena's ground-truth trajectories — closes the defense gap or merely teaches harness quirks; and whether open Octagon + challenge-warehouse runs by third parties reproduce the Sol/Opus keep-rate split outside the authors' 12-entrant set.
[1]Close
CyberArena's usable sentence is not "AI agents can hack." It is: once solo CTF benches stop separating the frontier, a live symmetric attack–defense score still can — and in the 466-match snapshot the residual is defense retention and restart speed, not another point of capture rate. Six months from now, either static attack-only leaderboards will still predict CyberArena order (thesis fails), or buyers will start quoting keep-rate and time-to-first-restart the way they now quote Pass@k.
[1]