Six accelerators feed one trainer
Modal's experiment today trains Qwen3.6-35B-A3B with asynchronous reinforcement learning on agentic coding, with up to 256,000 tokens of context. The tasks are the 2,700 repository tasks Xiaomi released with MiMo-V2.6, after history that leaked the reference fix was removed. The reward is 1 if the patch passes tests, otherwise 0. Evaluation is SWE-bench Pro V2 and its HARD-51 subset, and every checkpoint is scored in BF16.
The mixed samplers cover six machines: B200 and B300 in NVFP4, H100 and H200 in FP8 or BF16, and A100 and RTX PRO 6000 in BF16. The comparison run is BF16 on B200 only. The trainer stays BF16 on B200.
[1]
Uncorrected training collapses. Matching hardware only delays it
GRPO without a correction collapses on the mixed samplers soon after training starts. The B200-only run lasts longer and then collapses too. In a collapse the training reward falls, the sampler-trainer KL grows by 3 times or more, and most episodes end on repeated format errors. At step 40 of the mixed run, 62% of evaluation episodes stop calling the tool, at a median of 22 turns, against 57 turns for passing episodes. The last reply is empty in 98% of those failures.
Any single piece of the correction only delays the collapse. Only score centering plus masked importance sampling, with or without top-p, trains through to the end. Pass@4 on HARD-51 rises over roughly the first 100 to 140 steps, then levels off at about 65% to 75%. The full set moves from 89% to about 94%.
[1]Once the scores match, the extra cost is KL
With all three pieces, pass@4 on the mixed samplers stays close to the B200-only run. Average KL is about 4.7 times larger. For most of training, cost per update was about 15% lower on the mixed samplers. The authors say the setup was meant to test whether training holds, not to minimize cost.
[1]