Apple’s 8B online dense RL: decouple planning from grounding on live iOS, tie Gemini-3.1-Pro on UINavBench

In November 2026, Apple Machine Learning Research posted Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps (arXiv:2610.39264; Peitong Duan*, Khashayar Gatmiry*, Jeff Nichols, Emmanuel Abbe, Samy Bengio). The claim is specific: frontier VLMs can navigate UIs but are too expensive for everyday use; smaller models trained with SFT or offline RL often memorize static trajectories and fail to recover. Their answer is to decouple a Planning agent from UI grounding, spend expensive online RL only on planning, and train on hundreds of concurrent live virtual iOS devices with an LLM-as-a-Judge that supplies dense step-level rewards and a terminal reward. On UINavBench (real iOS apps), an 8B PPO checkpoint ties Gemini-3.1-Pro; on AndroidWorld—never used for the RL phase—success is nearly seven times the SFT baseline.

What happened

The architecture refuses to bind “what to do” and “where to tap” into one RL-trained policy. The Planning agent reads the screenshot and history, then emits a chain-of-thought current step plus a remaining plan; a Grounding model maps that step to primitives (Tap, Swipe, Text_Entry, …) and coordinates. The bet is that grounding is a largely deterministic map that SFT already solves well, while planning needs open-ended recovery—so expensive RL is reserved for planning and grounding stays frozen SFT (internal 3B; location swipes routed to Gemini-3.5-Flash). The action space also includes Ask_User, with Gemini simulating preference replies for underspecified tasks.

Rewards are not terminal-only. A dense judge (Gemini-3.5-Flash) scores each step as success / fail / overcomplete, penalising hallucinated progress and unproductive loops; a terminal judge (Gemini-3.1-Pro) uses the last three screenshots plus action history for success / fail / blocked_complete. Optimisation is PPO with step-level GAE: the dense channel discounts at γ_dense=1/2 while the terminal reward is carried undiscounted, to block reward hacking that lengthens trajectories for step credit. GiGPPO further conditions the critic on the nearest perceptual-hash neighbor return from sibling concurrent trajectories of the same task, because exact screen repeats almost never form useful subgroups. Infrastructure claims hundreds of concurrent live virtual iOS devices with snapshot rollback (up to 512 trajectories in the text).

How hard is the evidence

Table 1 pins the claim to checkable numbers. On UINavBench (116 tasks / 36 real iOS apps): SFT 16.7%; PPO Step 336 reaches 32.4%, tying Gemini-3.1-Pro and beating Gemini-3.5-Flash (28.4%) and Qwen3.8-27B (19.6%). On AndroidWorld (OOD; no Android in the RL phase): SFT 4.7% → PPO Step 336 32.8%—nearly sevenfold, the authors’ main evidence that the policy learned transferable navigation rather than iOS trajectory memorisation. iOSWorld is more modest: GiGPPO Step 336 hits 9.8% (SFT 4.5%), while most checkpoints remain far below frontiers (Gemini-3.1-Pro 39.5%).

Ablations are equally concrete: terminal-only PPO reaches only 21.6% / 19.8% on UINavBench / AndroidWorld at Step 336; dense PPO at Step 168 already hits 25.5% / 20.7%—summarised as dense signal roughly matching an extra epoch of terminal-only training. On UINavBench, the closest match to the training distribution, early GiGPPO checkpoints (75, 168) often catch later standard PPO, supporting the mechanism story that neighbor-conditioned value estimates accelerate convergence. Frontier models still sit much higher on AndroidWorld (Gemini-3.1-Pro 60.8%)—gains and remaining gaps share one table.

Steelman and gaps

Steelman: decoupling planning from grounding and spending online RL only on open-ended reasoning is an honest compute choice; dense step judges attack the classic sparsity of terminal-only credit; asymmetric discounting targets length-for-credit hacking; concurrent live-device infrastructure makes online interaction more than a toy. The related-work contrast with concurrent ClawGUI-RL is explicit: that line trains end-to-end on Android emulators at about 64 concurrent environments; this paper emphasises decoupling, neighbor signal routed through a critic, and hundreds of live iOS devices.

Gaps are equally sharp. iOSWorld uses synthetic apps plus heavy personalization and multi-app workflows that are OOD to training; the authors note later checkpoints colliding synthetic app names with real App Store titles or treating them as Safari web services. On AndroidWorld, frontier leads may partly reflect abundant public Android UI pretraining data versus scarce public iOS sets—so “ties Gemini-3.1-Pro” is primarily a real-iOS-board claim, not a cross-ecosystem crown. Judges and Ask_User replies both depend on Gemini-family models, raising reward–eval kinship risk; SFT CoT and remaining-plan targets were also Gemini-generated. Those are not footnotes—they bound how far the thesis travels.

Who pays, and which questions to ask

For products and platforms: if everyday agents are priced out by frontier per-step inference, the path may not be “a larger planner,” but whether you can get dense process supervision on live environments and spend RL on the layer that must recover. For training teams: the terminal-only versus dense PPO epoch contrast is executable discipline—without step progress signal, doubling update steps may not buy the same convergence. For eval communities: reading UINavBench, iOSWorld, and AndroidWorld together prevents mistaking “ties on real iOS” for “already tied on synthetic personalization or Android OOD.” For buyers and internal pilots: ask whether training is online, whether rewards are step-dense, whether grounding is jointly RL-trained, and whether judge models kinship-match the public eval stack.

What to watch in six months

The next falsifiable step is concrete: without Gemini-family dense judges—or with an independent process reward model—can an 8B decoupled agent still sit in Gemini-3.1-Pro’s band on UINavBench; and on synthetic personalization (iOSWorld) and Android OOD, can dense online RL keep closing the frontier gap rather than shining only on the real-iOS board closest to training. Reproducers will keep stress-testing concurrency claims—“hundreds of live virtual devices” collapses to emulator folklore if unavailable. The claim is not “small models already beat frontiers,” but that for everyday UI navigation, online dense rewards plus a planning/grounding split may track an affordable capability curve better than simply enlarging the planner.