要点
- Ai2 releases Old Ideas, Novel Problems with open Novelty Bench data and code to stress-test LLM novelty judges.
- Across six frontier models, tiny prompt edits swing accuracy by >50 percentage points on the same idea pairs (one case: 93.5%→40.9%).
- Retrieval and extra reasoning do not reliably help; two purpose-built novelty systems underperform a simple Claude Opus 4.6 baseline.
In the automation-of-science story, the job most often handed to a second model is deciding whether an idea is novel enough. Ai2’s 27 October 2026 study argues that this judge is far from reliable.
The paper Old Ideas, Novel Problems and the open Novelty Bench make a sharp claim: novelty evaluation is setup-fragile. If a few words flip the verdict, pipelines that use LLMs to filter AI-generated research ideas will treat noise as progress.
[1]Ai2 built an automated pipeline around ICLR 2026 peer reviews, mining passages where reviewers explicitly discussed originality rather than treating acceptance as a novelty proxy. A high-novelty pool has 154 human ideas from top-10% accepted papers whose reviewers positively identified novelty without dispute; a lower-novelty pool has 145 ideas from bottom-10% rejected papers with originality concerns and no positive novelty signal. A second setting, Human+Generated, pairs the same 154 human ideas with ideas from a bare LLM generator given only a research area and an instruction to propose something novel.
Six judges—GPT-5.1, 5.2, 5.4, Claude Sonnet 4.5, Opus 4.5, Opus 4.6—were tested while varying prompt, reasoning budget, literature visibility, idea formatting, and aggregation.
[1]Prompt wording dominated. On Human+Generated, telling GPT-5.4 that exactly one idea in each pair was novel yielded 93.5% accuracy; a simpler “which is novel?” prompt fell to 40.9%—below chance—and flipped more than half the pairs. The same prompt change moved Human-Only accuracy by only about seven points, suggesting fragility is worst precisely where AI-generated ideas enter the loop.
Annotation checks: polarity agreement on 47/50 review-mining cases; an expert preferred the human idea in all 30 audited Human+Generated pairs. Paper (arXiv:2610.02022), dataset, and code are public.
[1]Steelman: Human+Generated labels are weak by construction; a different task framing (“find a publishable direction”) might reduce prompt sensitivity; purpose-built systems may simply be under-tuned.
Still thin without full failure tables: retrieval corpus bias, and whether rewriting ideas into a shared structure removes style shortcuts for every model. Confidence in the instability finding is high; confidence that every lab workflow inherits the same swing is medium.
[1]For AutoDiscovery-style stacks, the binding constraint may be evaluator calibration, not idea generation. For funders and conferences, LLM novelty triage without a locked protocol systematically rewards prompt craft over contribution.
[1]Watch for locked-prompt, fixed-retrieval, human-spot-check novelty protocols—and whether generators can still shop for a softer judge once the protocol is fixed. If shopping stops working, automated discovery gains a credibility floor.
[1]