要点
- Collinear AI / HF (2026-11-03): CWE-bench v2 opens 100 Harbor tasks across 91 CWEs; programmatic scoring; held-out retained.
- Quality gates: realistic instructions, graded behaviors inferable, behavior-not-patch verifiers, starter fail/reference pass, functionality kept, anti-memorization of upstream patches; hand review.
- Gemini 4 Argon leads 67.8% pass@1 / 81.0% pass@4; Opus 5.5 67.3%/78.0%; 95 tasks solved by ≥1 model; leader still fails ~19% always.
- Cost Pareto: DeepSeek V4.1 Flash 57.0% at $0.04/rollout; Opus +~10pt at $1.83; Gemini at $3.92.
Via a 3 November 2026 Hugging Face post, Collinear AI opened CWE-bench v2: 100 public Harbor tasks spanning 91 CWEs. An agent gets a real open-source checkout plus a short instruction and must find and fix a weakness without breaking existing behavior; scoring is fully programmatic.
Thesis: open cyber-agent evaluation is moving from opaque private red-teams to reproducible task packs; even the leader still fails ~19% of the set on every attempt, so “vulnerability fixing is solved” does not match the scoreboard.
[1]What shipped
Each task was checked against six quality questions: realistic instruction, graded behaviors inferable from the instruction, verifier checks behavior not one patch shape, starter fails / reference fix passes, functionality preserved, memorized upstream patches fail. Tasks were hand-reviewed; a held-out set remains (contact cyber@collinear.ai).
Open-weight results update via PRs on the dataset page; a Space leaderboard adds API-only models. The public set, grader, and harness are open; Harbor can run cwe-bench@2.0 straight from the Hub.
How hard is the evidence
Of 11 evaluated models, Gemini 4 Argon leads at 67.8% pass@1 / 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% / 78.0%). Ninety-five tasks are solved by at least one model—so they are solvable—yet the leader still fails ~19% on every attempt.
Cost Pareto: DeepSeek V4.1 Flash (1017) hits 57.0% pass@1 at $0.04/rollout; Opus adds ~10 points at $1.83; Gemini edges ahead at $3.92. Programmatic checks plus a held-out set are harder than one private demo—but this remains a 100-task curated slice, not the internet’s vulnerability distribution.
[1]Steelman and gaps
Steelman: defenders must cover attacker surface exhaustively; packaging 91 CWEs as reproducible Docker tasks is closer to an auditable capability claim than blog red-team anecdotes.
Gaps: the public set can be trained against (held-out only partly mitigates); “realistic ticket” is still curator judgment; pass@k and dollar costs reflect author harness choices; find-and-fix ≠ exploit weaponization or supply-chain sabotage.
[1]Who pays, what to watch
Security vendors and model labs pay when marketing says “autonomous vuln fix”—point them at the long tail and $/point. Buyers and regulators can treat “reproducible harness + held-out” as a transparency floor.
Watch whether community PRs lift open weights, whether held-out rotates, and whether RL on the same environments shrinks the 19% tail.
[1]Close
The defensible sentence: a public, programmatic, multi-CWE agent repair benchmark makes cyber capability checkable—and today’s frontier still leaves about one task in five unsolved. That neither proves models are useless nor that 67% pass@1 is production autopatch; it falsifies the slogan “already solved.”
[1]