要点
- Named adopters include Xiaomi MiMo-V2-Flash (named MOPD), DeepSeek-V4, Moonshot Kimi K3, NVIDIA Nemotron 3 Ultra, Liquid AI LFM2.5, Mistral Large 4, among others.
- Three-stage recipe: general SFT, per-domain RL experts, then fusion where each prompt is scored by its domain teacher (usually reverse KL on the student’s tokens).
- TRL AsyncDistillationTrainer routes via teacher_id and teacher_server_urls; Open-MOPD math/code/IF experts on the Hub; HF Jobs h200x4 launcher example.
- In a 60-step TRL run with 40/40/20 prompts, IF token share fell to about 1-3%; math/code moved toward teachers, IF barely moved.
- Eval table: MixSFT AIME24/25/IFEval 17.5 / 19.2 / 67.5; 60-step MOPD 24.2 / 22.5 / 73.0; Open-MOPD final 19.2 / 25.0 / 75.8.
Hugging Face’s TRL blog, by Sergio Paniego, is not a soft “distillation is popular” note. It is a systems claim about how labs turn many domain RL experts into one model, and what breaks when you copy that recipe without watching tokens. Xiaomi named Multi-Teacher On-Policy Distillation for MiMo-V2-Flash in January 2026. Since then DeepSeek, Moonshot, NVIDIA, Liquid AI, Mistral and others have used variants. TRL’s AsyncDistillationTrainer now lets outsiders run the fusion stage with served teachers and a teacher_id column.
[1]What shipped
MOPD, strictly, is the last stage of a three-stage post-training recipe, though labs often use the name for the whole path. Stage one is general SFT on mixed data. Stage two trains one RL expert per domain from that checkpoint. Stage three puts a student, often the SFT checkpoint again, on prompts from every domain. The expert for that domain scores the student’s answer token by token. The student is trained toward that teacher’s distribution, commonly with reverse KL. The student learns from its own samples. The signal is dense per token rather than one reward per answer. Each prompt goes to one teacher.
Labs differ on how much of the teacher distribution they use. Nemotron 3 Ultra and MiMo-V2-Flash use the teacher probability for the token the student wrote. Open-MOPD compares the top 16 tokens. DeepSeek-V4 and Mistral Large 4 use the full vocabulary distribution after single-token feedback proved unstable for DeepSeek. Teachers are usually the same size as the student and share a tokenizer.
The post’s adopter table runs from Xiaomi MiMo-V2-Flash through DeepSeek-V4, Moonshot Kimi K3 (nine experts by domain and reasoning effort), NVIDIA Nemotron 3 Ultra (more than ten teachers, two MOPD rounds), Liquid AI LFM2.5, Mistral Large 4 (teachers by reasoning effort rather than domain), and related releases. NVIDIA also put five Nemotron teachers on the Hub. Open-MOPD releases the full recipe and checkpoints that the TRL example uses.
In TRL, AsyncDistillationTrainer takes a map of teacher_server_urls and a teacher_id on each row. Generation, teacher scoring, and training run asynchronously. The shipped example uses Open-MOPD’s SmolLM3-3B MixSFT student and GRPO experts for math, code, and instruction following, with prompts interleaved 40/40/20. With beta=1.0 reverse KL, TRL matches the single-token teacher logprob style of Nemotron and MiMo. A launcher.py fits student vLLM, three teachers, and training onto four H200s via HF Jobs.
[1]Evidence on the table
Primary evidence is the 3 November 2026 Hugging Face blog, including the adopter table, the three-stage diagram, the AsyncDistillationTrainer code path, the HF Jobs command, and the 60-step Trackio-style metrics. In that short run, instruction following began with a fair token share then fell to about 1-3% once math and code answers grew long, while teacher_jsd for math and code fell and IF barely moved. The evaluation table compares MixSFT, the 60-step checkpoint, and the paper’s final Open-MOPD model on AIME24, AIME25, and IFEval. Code evaluation was not run. The authors point to the Open-MOPD paper for the imbalance diagnosis and a fix. Independent multi-lab head-to-head comparisons of TRL MOPD are not in the post.
[1]Steelman and gaps
Steelman. Mixed RL in one run lets domains fight inside every batch. Training separate experts then fusing them is a rational answer, and on-policy token-level distillation reduces the train/serve mismatch of plain SFT on teacher text. Open teachers and AsyncDistillationTrainer matter because most labs only ship the final merged model.
Gaps. The 60-step run is about 1% of the paper’s training length, so absolute scores are a smoke test, not a claim that TRL matches production MOPD. Prompt rebalancing to 40/40/20 still left IF starved on tokens, which is the point. Reverse KL with top-k=1 does not reproduce Open-MOPD’s top-16 comparison. Full-vocabulary teachers need more bandwidth. Anyone without same-tokenizer RL experts cannot plug in cleanly.
[1]Who should care, and what to watch
Open post-training teams, labs that already own domain RL experts, and eval designers who only watch prompt mix ratios should treat teacher_token_frac as a primary dashboard metric. Watch over six months: whether more labs release expert checkpoints; whether DistillationTrainer lands for synchronous runs; whether public recipes adopt Open-MOPD-style reweighting by default; and whether IFEval-style short domains stop lagging when token share is forced closer to prompt share.
[1]Close
The durable claim is not that distillation is new. It is that multi-teacher on-policy fusion is now a shared frontier recipe you can run in TRL, and that the silent failure mode is token share, not prompt share. Six months on, either open MOPD runs publish per-teacher token fractions beside benchmark tables, or short domains will keep looking “trained” while barely updating.
[1]