Mistral Large 4: falsify the equal-perplexity agentic gap—not a parameter slogan

Agentic coding and cyber tasks need long-range dependencies, while serving KV cost pushes stacked memory-saving tricks. Mistral Large 4’s report narrates: save memory first, discover that “fine-looking” LM metrics under-predicted long-horizon agentic results, surgically restore long-range attention, then re-sparsify to reclaim cost.

Falsifiable read: if independent reruns no longer show systematic SWE-bench-class gaps across attention variants at matched perplexity/MMLU-class scores, or full-scale YOCO/DSA ablations cannot support the report’s “agentic penalty” causal story, discount the claim that architecture choices at equal LM loss materially decide long-horizon agentic ability.

Hard facts (alphaXiv tech-report page / abstract & overview, submitted 3 Nov 2026):

• Scale & benches: 52B active, 1T total multimodal MoE; table scores FrontierCode 2.0 19.7%, Terminal-Bench 4.0 41%, Artificial Analysis Cyber Index 50.9%.
• Early memory stack: ragged MHLA (full attention every 4th layer; 512-token SWA elsewhere) + CLA + YOCO; ~16× KV vs full-attention baseline; YOCO two-stage prefilling ~1.7× faster at 1M context. Common pre/post-train checks acceptable; later SFT weaker on long-horizon agentic tasks.
• Figure 5: MMLU Redux vs monorepo perplexity stays comparatively tight across variants; SWE-bench Verified vs validation perplexity spreads at similar perplexity—authors describe an architecture-imposed agentic penalty at comparable LM loss; knowledge evals did not show the same gap.
• Surgery & DSA: remove SWA limits and CLA; keep YOCO at full scale (surgery+midtraining favored retaining it at larger test scale, but full phase-2+midtraining schedule was not rerun). DSA indexer selects top 2048 keys; ~15.1B tokens indexer warmup with frozen full-attn distill, then ~198B joint sparse training. Sparse stage is framed as long-context cost control—not a demonstrated agentic gain over full attention.
• Serving: calibrated mixed-precision NVFP4 checkpoint 606 GB, said to match FP8 on evaluations; deployable on four GB300 GPUs. Optimized decode context parallelism and DSpark raised interactivity ~51%/90% at eight concurrent requests vs baseline (serving speed, not fewer generated tokens).

Evidence strength: first-party tech report + named benchmark scores + controlled-scale ablations and figures. 19.7%/41%/50.9% are final-model table scores; the report does not attribute the gain to DSA. Fig. 5 compares attention variants—it shows perplexity/knowledge missed an agentic dimension, not that attention is the sole cause. Keeping YOCO is a qualified choice for this model; authors admit the full-scale recovery schedule was limited.

Steelman: if hard serving memory forces stacked KV savings, and ordinary LM/knowledge metrics are blind to long-horizon tool use, steering architecture with SWE-bench-class tasks fits product failure modes better than perplexity alone; DSA as a cost backfill after restoring full attention is coherent.

Strongest steelman: long-horizon agents can diverge under “lookalike” loss curves; pairing an executable-task synthesis pipeline (seed → cheap checks → agentic verifier → difficulty calibration) with architecture surgery treats data and attention as co-equal.

Steelman risks: ablations mostly at limited compute; full-scale YOCO recovery schedule incomplete; final score gain not attributed to DSA; authors note agents remain fallible and some domains need post-hoc filtering—external reproduction is expensive.

For open-weight and inference teams, watch whether independent attention ablations reproduce “matched perplexity, divergent SWE-bench”; whether the NVFP4 606 GB / 4×GB300 claim holds under third-party loads; and whether the DSA indexer warmup/joint schedule is publicly reproducible. Until then, read the report as an architecture decision log and serving recipe—not a single leaderboard verdict.

On a six-month horizon, the falsifiable questions are whether long-horizon agentic evals become a default architecture gate, and whether sparse attention remains an efficiency patch rather than a substitute for measuring agents. If third parties reproduce “similar loss, divergent agent scores” and the NVFP4 serving claim under a public protocol, ML4’s narrative holds; if only the score table and OG image remain, it stays launch collateral.

【No.2】