METR on Anthropic: falsify 1.9x AECI, not a transparent “AI acceleration ledger”

METR published an independent assessment of Anthropic on 2026-11-05. From 2026-08-12 to 2026-10-29, staff ran a high-access review of pretraining, RL, compute by domain, internal benchmarks, AI speedup analyses, and staff interviews to judge how far AI is accelerating Anthropic’s own AI R&D.

The checkable public number is that since October 2025, Anthropic internal models’ AECI scores (Anthropic’s Epoch Capabilities Index variant) have grown at about 1.9x the prior rate, 95% CI 1.5-2.8, and the team is confident the acceleration was not from faster growth in training compute. Most METR staff on the project now believe that as of October 2026, Anthropic is improving models on well-specified tasks at roughly twice the March 2024-October 2025 historical rate, and they guess that a majority (perhaps ~80%) of the current acceleration comes from extensive AI use in R&D.

The same post’s hard limit: Anthropic redacted a majority of the evidence METR found most informative for those conclusions. METR says that blocks the public from re-judging attribution and bottlenecks, so readers must lean on METR’s judgment more than is ideal. The news core is “1.9x AECI + compute-scale ruled out + most informative evidence redacted,” not reading “~2x / AI-majority” as a transparent ledger.

Hard facts (METR post, accessed 2026-11-06; dated 2026-11-05):

• Window: 2026-08-12 to 2026-10-29; high-access evaluation at Anthropic.
• AECI acceleration: about 1.9x since October 2025 (95% CI 1.5-2.8), roughly two years of historical benchmark progress in the past year.
• Compute: team confident acceleration was not due to faster scaling of training compute.
• Subjective risk view (METR): ~2x historical pace on well-specified tasks as of October 2026; would not be surprised by ~1.3x or ~3x; extra uncertainty on transfer to broader, less well-specified tasks.
• Attribution: late-2025 acceleration likely influenced by non-AI factors (many details redacted); for “current” October 2026 acceleration, METR guesses AI is a majority (perhaps ~80%), with substantial uncertainty; finer cumulative/instantaneous AI-effect estimates were redacted.
• Bottlenecks: no clear bottlenecks seen that would block further capability progress in the next 12 months from algorithms, training compute, experiment compute, or data, but understanding of bottlenecks changed several times; low confidence.
• Mode of work: told acceleration is mainly via AI assistance rather than fully autonomous contributions, with some signs autonomy may grow.
• Redaction: Anthropic could redact non-public information; METR could disclose whether rights were used and whether findings depended significantly on redacted material. Anthropic said redactions aimed to remove confidential information that could accelerate other AI developers. METR says most of the most informative evidence was redacted; the report does not achieve “transparency about evidence.”
• Limits (METR’s own): possible underestimation (further recent growth-rate increases; more R&D-task automation; AECI sometimes understating real-world impact) and overestimation (non-AI factors; AECI’s focus on verifiable tasks; unusually slow early 2023-2025 growth).

Terminology: METR defines acceleration as a ratio of growth rates; 2x acceleration means one year delivers two years of progress at the old rate.

Evidence strength: a public summary of a high-access independent assessment, not a raw data room. The 1.9x AECI figure with CI, and “not faster training-compute growth,” are quantitative claims readers can cite from the post. “~2x / AI ~80%” is METR’s subjective synthesis after seeing material that includes heavy redaction. Interviews and self-reported uplift are flagged by the authors as hard to interpret.

The public falsification path is narrow: outsiders cannot see the redacted tables or pipeline estimates, so they cannot rebuild the attribution split. What remains: track whether later public Anthropic models’ reproducible-benchmark progress points the same way as “two years of progress in one”; watch for a second lab opening a comparable read-only review; read METR’s process table of what could and could not be discussed to see which conclusions explicitly lean on redacted evidence.

Steelman: if AECI’s method is stable and the “not compute-growth acceleration” exclusion holds, the quantitative sentence “benchmark progress sped up sharply since October 2025” can stand. Booking most of that speedup to AI assistance in R&D remains a judgment under redaction, not a public ledger the reader can recompute.

Strongest steelman: frontier labs rarely let outsiders deep-read pretraining, RL, compute, and internal benches. METR’s ability to publish an AECI acceleration with a confidence interval, while admitting redaction and internal disagreement, is denser information than a company blog alone. Putting “maybe 1.3x-3x” and “bottleneck views flipped several times” in the summary also cuts false precision.

Steelman risks: most of the most informative evidence is invisible, so the public cannot referee “AI ~80%” versus “non-AI factors in late 2025.” AECI may overfit verifiable-task distributions, or misread recent acceleration because points are sparse and lag research. Not seeing bottlenecks is not proof none exist; the authors warn their view flipped more than once. “Fully autonomous contributions may grow” is a signs-level note, not a measured timetable.

So a headline that says “Anthropic is already ~2x AI-accelerated” over-translates the post. The steadier reading is: the public quantitative claim is 1.9x AECI (1.5-2.8) with compute-scale acceleration ruled out; “~2x / AI-majority” is METR’s judgment under redaction.

For policy and safety communities, the next six months are whether a second comparable high-access assessment appears, whether Anthropic redacts less of the most informative evidence, and whether public benchmark slopes keep diverging from the 2024-2025 history. For other labs, the pressure is whether to allow similar external read-only access; refusal looks like an information gap, acceptance trades against leakage and redaction fights.

For investment and procurement narratives, split “publicly citable numbers” from “judgment-dependent attribution” so METR’s synthesis is not pasted in as an audited productivity multiple. For Anthropic staff and rival researchers, “assistance-first, autonomy signs rising” hints process may change before fully autonomous agents do, but that still needs internal metrics in public before it travels.

METR’s assessment should be judged on the public 1.9x AECI interval and the “not training-compute growth” exclusion, plus the process section’s explicit redaction dependence, not on treating “~2x / AI-majority” as a transparent ledger. If the interval and exclusion hold, it is a substantial high-access summary. Attribution shares and “no bottleneck” calls should stay labeled as judgment while most of the most informative evidence remains blacked out.

Also: METR says both sides have significant uncertainty about how to synthesize the evidence. Readers who only repeat “2x acceleration” drop the error bars and redaction warning the authors printed.

【No.2】