The shelf life of LLM findings: what’s reproducible is the decay curve—not “everything expires”

In 2023, Walters & Wilder found 55% of ChatGPT (GPT-3.5) literature-review citations were fabricated; the same prompts rarely invent citations on GPT-6.1-Sol or Claude Sonnet 5.5—though exceptions remain. Queen and Zou systematize that “true yesterday, still true today?” question.

Falsifiable read: if “LLM empirical claims have short shelf life” holds, capability failures should decay with release date while preferences persist; if a new panel flips categories, or reproduction anchors systematically miss, lower the extrapolated shelf-life weight.

Hard facts (project site, 2 Nov 2026): 50 claims spanning sycophancy, jailbreaks, long context, cultural bias, and more; trends fitted by model release date; expiration zone e≤0.2. Counts: 22 expired, 17 weakened, 8 persist, 3 strengthened. Strengthened: judges’ self-preference (C28), praise following user preference (C17), nonuniform random numbers (C29).

Method: standardize paper metrics to relative effects; reproduction checks (43 of 48 numerical claims within magnitude/uncertainty tolerance); then fit curves (constant/linear/exponential/logistic through the reproduction, lowest AICc) on later panel models. Authors stress: findings are observations about particular models at particular times—not fixed properties of LLMs.

Evidence strength: primary research site with category rules, timeline, citation case study, and reproduction detail. Citations: GPT-3.5 Turbo reproduction ~59% fabricated (near paper’s 55%); Claude Sonnet 5.5 ~1%, GPT-6.1 Sol 0%; yet Qwen3.7-Flash fabricated 69% in this test. Authors argue conference desk-reject policies keyed on hallucinated references (citing ICLR 2026 / ICML 2026 grounds) may be a poor primary detector of AI writing without oversight.

Steelman: communities bake old failure modes into policy and trust narratives; systematic retesting updates priors. Beyond: only 50 claims (wide CIs); early expirations rest on few 2024 models; substitutes for retired originals; default API configs without a common reasoning budget.

Strongest steelman: capability failures have right answers developers can train against, so they expire faster; preferences and judgment are harder to scrub with a single supervised signal—matching “five of nine reasoning claims expired; preference claims more durable.”

Steelman risk: weakened ≠ expired (e.g., two-hop: Claude Sonnet 5.5 still ~10% wrong when both hops are correct; paper reported 43% for ChatGPT); panel-wide expiration ≠ every model; mixing closed and open models in one fit can hide provider-specific paths.

For researchers and reviewers: claims should bind model, version, and date, with prompts and code for future reruns. For policymakers: compliance metrics centered on old failure rates need recalibration. For users and firms: trust lag affects adoption—but update with data rather than freeze on 2023 narratives.

Do not read “mean 15.7 months to expiration” as a half-life law for all LLM papers: it describes the expired subset and excludes not-yet-expired claims. The portable discipline is: report model identity and date, keep reproducible artifacts, treat empirical claims as time slices—then retest on newer models.

【No.2】