Apple's amortized MIPS: RAG speedups may come from knowing your queries

On 5 November 2026, Apple's machine-learning research blog highlighted a retrieval paper headed to ICML 2026. The object is maximum inner product search, the workhorse behind recommenders, search, and RAG. Approximate indexes usually carve the key set into clusters and route by centroids. Apple's authors argue that production systems already know a lot about their query distribution, and that RAG queries look unlike the long passages they retrieve. The thesis follows. When queries are stable and keys are fixed, amortizing MIPS into small learned routers can beat another purely geometric index tweak.

What shipped

The mathematical hook is that the map from a query to its best inner product over a key set is that set's support function: convex, and with a gradient that recovers the winning key. The team trains small networks offline on sampled queries paired with exhaustive MIPS answers. SupportNet approximates per-cluster support scores so routing asks how good the best key inside a cluster is, not only how close the centroid is. KeyNet predicts the best key vector directly and can rewrite the query before a standard IVF index runs.

Both are regression models over the query distribution that matters. Reported SupportNet and KeyNet families (XS to L, about 1% to 20% of database size, depth 16) plug into existing IVF machinery rather than replacing the whole stack. The blog's guidance is practical: use KeyNet when raw speed to the best key matters; use SupportNet when you also want calibrated scores or when small models must stay cheap; large systems may want SupportNet over clusters plus KeyNet inside them.

Evidence on the table

The main public numbers use Natural Questions with roughly 2.6 million passages and 128 clusters. For routing, a small SupportNet reaches 95% routing recall about 1.9× faster than centroid routing (1.74 ms versus 3.36 ms), with larger nets pushing higher recall at similar latency. At a fixed scan budget of about 5% of the database, learned routers recover the true partition about 90-93% of the time versus 87.5% for centroids. For search, KeyNet-mapped queries hit 99% recall up to roughly 8× faster than raw queries (0.23 ms versus 1.8 ms), and reach 99% recall after scanning up to about 11× fewer keys in the quality-only view. Example top-3 galleries show KeyNet changing which passages surface for the same question.

These are Apple-reported figures on standard academic data, not a multi-tenant production A/B. The post says trends held across several datasets studied, without dumping every table in the blog form. Training assumes a fixed key database and a characterized query source. Mild query shift is claimed to be tolerable; major drift is the obvious break glass.

Steelman and gaps

Steelman. If your corpus churns hourly, offline amortization becomes a tax. If queries are adversarial or open-world, the "known distribution" premise thins out. Neural routers add failure modes and ops surface that FAISS-style centroid code does not. Leaderboard latency plots can hide training cost, memory for the nets, and tail latency under concurrent load. Centroid IVF plus better quantization may close part of the gap without learned components.

Those objections define the deployment filter. They do not erase the geometric point Apple is pressing: RAG keys and RAG questions are not exchangeable clouds, so query-agnostic geometry leaves free accuracy and latency on the table when you already have logs of real questions.

Who should care, and what to watch

RAG platform teams with stable corpora and logged query traffic should treat amortized routers as a first-class experiment beside embedding upgrades. Search infrastructure groups gain a reason to keep IVF and add a learned front door instead of a full rewrite. Academic MIPS work gets a convex-analysis hook that is unusually clean for systems papers.

Next six months: whether open reimplementations match the NQ latency-recall curves; how often keys must be rebuilt before nets go stale; and whether any large consumer assistant cites SupportNet/KeyNet-style amortization in production retrieval notes.

Close

Apple's useful sentence is not that neural nets can replace indexes. It is that RAG retrieval is a place where knowing tomorrow's query distribution is worth a one-time training bill. Six months out, either production stacks show the same recall-at-latency shift on their own logs, or amortized MIPS stays a neat ICML result that generic IVF continues to out-ship.