要点
- Hugging Face blog (2026-11-02, sirluk) runs a memory-matched distillation follow-up to “Sliding-window beats linear attention”: four students from Qwen3-4B-Instruct, each layer budgeted as 576 cached tokens of KV.
- On short-answer benchmarks SWA (learned sink) leads Bifocal by ~0.7–1.5 points; authors agree SWA+sinks is a strong cheap baseline.
- On AIME/HMMT long-answer math (~7,600 tokens/answer), Bifocal solves ~1.8× as many of the 96 long-answer problems; correct answers >4,000 tokens: 66 vs 26 for SWA.
- Almost-all-state Bifocal (window 64) collapses to 4.2% math vs 15.5% — exact recent recall still required. In vLLM, Bifocal decode ≈ SWA ~195 tok/s out to 128K prompts.
When reasoning models write long answers, the KV cache grows linearly with tokens: the post pegs Qwen3-4B at ~144 KiB per token, so a 10,000-token answer needs ~1.4 GiB. Fixed memory leaves two levers—forget older tokens (sliding-window attention) or compress the past into a fixed state (linear attention / Mamba / GDN / mLSTM).
A recent paper argues SWA with attention sinks is enough on its own, beating most linearized models on several benchmarks. Lukas (sirluk) on Hugging Face runs a narrower, falsifiable follow-up: at matched memory, SWA is genuinely strong on short answers; once the model must generate multi-thousand-token reasoning chains, a Bifocal window+linear mixer pulls ahead.
[1]What shipped
All 36 attention layers of Qwen3-4B-Instruct are replaced. Four students share the same ~4.1B-token distillation recipe and a per-layer budget equal to 576 cached BF16 KV tokens (linear states counted via FP32 recurrent size). Variants: 576-token SWA with learned sinks; 572-token SWA + 4 StreamingLLM sink tokens; Bifocal (448-token window + one mLSTM state per KV head ≈128 tokens); and Bifocal with window 64 (state per query head ≈512 tokens).
Bifocal mixes mLSTM and SWA with a per-head, data-dependent gate—sharp near, lower-resolution far. The SWA baselines are intentionally strong: same training budget as linearization, not training-free masks.
[1]How hard is the evidence?
Short answers (GSM8K / IFEval / HumanEval-class): SWA leads by ~0.7–1.5 points—prompt+response mostly fit the window; SWA also stays closer to the teacher architecture.
Hard math: AIME 2024/2025 and HMMT 2025 (30 problems each, 16 samples, pass@1). Mean answer length ~7,600 tokens; the window covers ~8% at any step. Aggregate Bifocal 15.5% vs SWA 13.8% (paired bootstrap 95% CI ~0.0–3.4 pts). By length: both ~59% on short-answer problems; on 96 long-answer problems Bifocal solves ~1.8× as many; correct rollouts >4,000 tokens: 66 vs 26. Window-64 Bifocal collapses to 4.2% math. In vLLM on a Qwen3.5-4B-shaped stack, Bifocal and SWA both hold ~195 tok/s decode to 128K; Bifocal prefill is ~63–83% of SWA’s rate.
[1]Steelman and gaps
Steelman: SWA+sinks is cheap and should be reported; many linear models in generative/long-context tables were retrofitted on 20–600M tokens; “long context” often means long inputs with short answers; log-likelihood suites never read the model’s own writing—so a 64-token window recovering 99% proves little about distant memory.
Gaps: the overall math gap is statistically thin until stratified by length; teacher/data fixed (Nemotron mix)—no claim about larger models or native linear pretraining; sink-token SWA evaluation bugs on over-cache prompts led authors to report math only; GPQA dropped for truncation. The “stacked windows ≈ L×W” theoretical span vs ~1.5-window effective range is mechanistic argument, not an independent replication.
[1]Who pays / what to watch
Serving and long-chain training pay: if production answers run thousands of tokens, choosing pure SWA from short-answer or multiple-choice tables can lose accuracy once the KV budget is binding. Builders should publish memory-matched comparisons and put generation length into the eval design.
Over six months, watch whether labs re-run “SWA vs linear” on long-generation sets; whether gated hybrids land in mainstream serving kernels; and whether native hybrid pretraining changes the same conclusion.
[1]Close
“Sliding window beats linear attention” holds for short answers and likelihood suites, and belongs as a baseline. If your bottleneck is KV under long self-generated reasoning, the matched-memory evidence supports a narrower sentence: the window keeps the recent past exact; the linear state carries a summary past the window—and spending the same budget on both is what separates them on long-answer math.
[1]