A preprint led by a Zhejiang University team, posted to arXiv on September 17 under number 2609.20784, proposes RetireOPD (Self-Retiring On-Policy Distillation), which uses an adaptive retirement mechanism to solve a distillation problem in multi-turn agent reinforcement learning: once a student's discrepancy with its teacher stops shrinking and the student reaches a target fraction of the teacher's success rate, the student drops teacher supervision and trains with RL alone.
[1]The problem is reward density. When RL trains a multi-turn agent, each trajectory earns a single scalar reward at the end, and the signal is sparse in between. Self on-policy distillation (OPD) supplies dense token-level supervision from a self-teacher that holds privileged task skills - information the student cannot see, such as environment internals or extra planning steps - letting a skill-free student internalize those skills. The paper reports two findings that undermine this recipe. First, privileged information alone does not always make a teacher reliable: a teacher can know more and still supervise poorly. Second, the benefit of teacher supervision is stage-dependent rather than constant - early training gains from it, later training can be dragged by it.
RetireOPD works in two steps. It first optimizes a decoupled, skill-conditioned teacher with environment rewards, so the teacher is trained to be useful before it is used. It then trains a skill-free student jointly with RL and OPD. Instead of following a predefined distillation schedule, its Adaptive Retirement lets the student decide when to graduate: when the student-teacher discrepancy stops shrinking and the student reaches a target fraction of the teacher's success rate, the teacher is dropped and training proceeds with RL alone. The mechanism is driven by the measured gap rather than a fixed schedule, which the authors argue avoids both premature abandonment of supervision and indefinite dependence on it.
The results are concrete. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over the RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting - the student ends up outperforming the teacher that supervised it. ALFWorld is a text-based household task environment where an agent follows instructions across rooms and objects; WebShop is a simulated shopping environment. Both reward long sequences of tool use and decision making, which is where sparse-reward agentic training typically struggles.
The caveats are worth stating plainly. These numbers are self-reported by the paper and not independently replicated, and the preprint has not been peer-reviewed. The retirement threshold, a target fraction of teacher success, is a hyperparameter set by hand. The evaluation environments are text-interaction tasks; vision or long-horizon agent settings are not covered, and the authors do not report compute costs for the teacher-student pipeline. Adaptive graduation settles when supervision exits, not whether the supervision was sound - and that second question remains open in agentic RL. The idea worth watching is not the distillation itself but the retirement rule: letting the training signal turn itself off when it stops paying.
[1]