要点
- Meta Engineering says Catching JiTTests is in production: unlike LLM/agent code review, it emits executable tests that show a behavioral difference introduced by a diff.
- In H1 2026 it helped catch 74 serious (SEV) privacy bugs at the diff stage and coincided with an ~20% drop in privacy SEVs; 589 engineers engaged (~64% of outreach) with ~60% true positives.
- The stack pairs dodgy-diff and intent-aware generation, an LLM-as-Judge ensemble plus rule-based RubFake, then ranks whether to reach out — human-on-the-loop, not human-in-every-loop.
On 29 October 2026, Engineering at Meta reported that Catching Just-in-Time Tests (JiTTests) has moved from research prototype into production infrastructure. The authors contrast LLM/agent code review — faster versions of a human reading a diff — with tests that fail on purpose and demonstrate a concrete behavioral difference.
The falsifiable claim is narrow: at Meta’s scale, executable failing tests plus human-on-the-loop outreach catch severe bug classes that review-style LLMs miss, while cutting per-item engineer load. If the production story is only prettier review prose, or false positives swamp teams, the claim fails.
[1]The system runs in five stages. Overnight on spare capacity it targets prior-day diffs scored as risky (Diff Risk Score–like). Two generators run: dodgy-diff treats every change as a mutant to enlarge the weak-catch pool; intent-aware infers intent and risks, builds mutants for those failure modes, and harvests tests that fail on the real diff.
An LLM-as-Judge ensemble then scores weak catches using test code, failure traces, the parent/child diff and inferred intent; RubFake supplies deterministic rule judgments. Ranked candidates may reach engineers. Humans set success criteria and watch aggregates — not every output. The FSE 2026 paper covers the initial deployment.
[1]Hard numbers in the post: 589 engineers (64% of outreach) engaged, reporting ~60% true positives and actionable findings; in H1 2026 JiTTests helped catch 74 serious privacy bugs at the diff stage alongside an ~20% reduction in privacy SEVs. Mechanistically, review LLMs accelerate a human reading path; JiTTests spend more compute for runnable failure evidence.
Limits: the 20% drop is contemporaneous, not a published causal trial; the 60% TP rate depends on engineer feedback definitions; “catches classes review misses” is argued from mechanism and production narrative, not a public head-to-head raw table. Confidence: deployment and SEV counts high; causal magnitude and portability medium.
[1]Steelman one: executable tests cost more — does overnight batching slow merges? Meta’s answer is risk-targeted diffs plus ranked outreach. Steelman two: 60% TP still implies noise; a loose on-the-loop policy can miss severe bugs. Steelman three: privacy SEV declines may also reflect other privacy infrastructure (Meta’s own recent PrivacyLib posts), so single-cause credit is unsafe.
Another gap: outsiders get impact summaries and architecture, not a reproducible open pipeline; orgs without risk scoring and spare capacity will see different cost curves.
[1]For platform companies, the piece reframes AI code review as an evidence-shape question: opinion text, or a test that goes red? For privacy/security teams, catching SEVs at the diff stage means users never see the failure. For vendors, human-on-the-loop — policy and aggregate monitoring over per-item babysitting — may be a more scalable product than yet another review bot.
As agentic development speeds shipping, testing that cannot keep up becomes a risk amplifier. The generalizable shape: define success early, let AI execute, intervene at policy level.
[1]Watch over six months: whether TP and outreach rates hold across more code domains; whether privacy SEV declines remain attributable after other interventions; and whether external teams can keep false positives manageable when copying executable-fail + ranked outreach. If executable differential tests keep catching SEVs that review-style LLMs miss, the thesis holds; if cost or noise force a retreat to review-only, rewrite it.
[1]