要点
- Refit = copying updated policy weights from training GPUs to disaggregated inference GPUs each step; layouts differ (TP/PP/EP/DP/context).
- Gather-and-broadcast rebuilds full tensors then broadcasts; logical payload ≈ 2× weight bytes (Kimi K3 ~11.2 TB, Qwen 3.8 2.4T ~9.6 TB at BF16).
- Reshard moves only shard intersections; NCCL M2N Reshard open-sourced; NVLink-domain grouping avoids duplicate network copies.
- NeMo RL hybrid: Reshard on feed-forward projections (~95% of params), gather-and-broadcast on small weights for coverage.
- Benchmarks: H100 Qwen3 235B 6.4s→1.9s; DeepSeek v3 14.9s→2.6s; GB200 DeepSeek 6.1s→2.3s; Nemotron Ultra 8.2s→2.3s; Qwen3.8 2.4T microbench 4.3s.
On 6 November 2026 NVIDIA’s developer blog argued that reinforcement-learning post-training pays a growing tax every time training GPUs must refresh inference GPUs with new weights. Soft framing is another NCCL feature. The sharper claim is operational. As open policies approach multi-trillion parameters, gather-and-broadcast refit already implies on the order of ten terabytes of logical traffic per step, so the scarce resource is how you move shards between mismatched parallel layouts, not only how you sample rollouts. If Reshard’s 2.7×–5.7× NeMo RL speedups hold when layouts churn under failure recovery, the claim holds. If teams cannot describe weights as DTensors, or hybrid paths re-inflate the payload, it shrinks to a microbenchmark.
[1]What shipped
In disaggregated RL, inference and training use different parallel maps. Classic refit all-gathers each parameter on trainers, broadcasts the full tensor to inference ranks, then each rank keeps its shard. That is simple and wasteful: destinations receive bytes they will discard.
Reshard reframes the job as moving one distributed tensor into another distribution. Source and destination DTensor descriptions (global shape, device mesh, placements) define the call. NCCL M2N Reshard computes shard intersections, then maps transfers onto NVLink domains so each contribution crosses the network once and replicas finish assembly over NVLink.
NeMo RL enables the path with refit_transport: nccl_reshard when generation is not colocated. Because every layer type is awkward to cover, the reference design reshards dominant feed-forward projection weights and keeps gather-and-broadcast for smaller tensors.
Evidence on the table
Table 2 is the hard core. On DGX H100, Qwen 3 235B (TP4×PP8×EP16 → TP8×DP16) falls from 6.4 s to 1.9 s. DeepSeek v3 671B (PP16×EP16 → TP32×DP8) falls from 14.9 s to 2.6 s. On DGX GB200 the same DeepSeek layout falls from 6.1 s to 2.3 s; Nemotron Ultra v3 from 8.2 s to 2.3 s. A refit-only microbenchmark quotes 4.3 s for Qwen3.8 2.4T BF16.
Payload arithmetic is stated without mystery: BF16 weight bytes times roughly two for all-gather plus broadcast. For models already near 5 TB of weights, that is ~10 TB logical per refit, every training step.
[1]Steelman and gaps
Steelman. Disaggregated RL is already how large labs buy efficiency. Communication libraries should absorb layout math so frameworks only emit DTensor metadata. Open-sourcing M2N Reshard lets non-NVIDIA stacks compare. Hybrid coverage is an honest engineering compromise rather than a claim that every edge case is solved.
Gaps: numbers are NVIDIA-on-NVIDIA (NeMo RL, NCCL, DGX). Speedups are refit latency, not end-to-end tokens per dollar. Feed-forward dominance (~95%) is asserted for “latest LLMs” without a public per-model table here. Runtime membership changes still destroy/recreate communicators in NeMo RL; shrink/grow is future work. Competing frameworks may already use custom P2P paths that narrow the gap.
[1]Who should care, and what to watch
Teams training large policies on separate inference pools should treat refit seconds as a first-class budget line. Cluster architects should ask whether NVLink domain topology is visible to the collective. Open-weight labs pushing past trillion-parameter RL care whether 4.3 s class refits stay real under mixed precision and expert parallelism.
Over six months watch: third-party frameworks wiring nccl_reshard or equivalents; published end-to-end RL step breakdowns that isolate refit; whether NCCL shrink/grow lands so reshard survives node failure without full rebuilds.
Close
NVIDIA’s post is easy to misread as “faster collectives.” The durable sentence is that RL at frontier weight sizes is becoming a distributed-systems problem of minimum necessary bytes between two live layouts. Six months on, either Reshard-class paths show up in step-time reports outside NVIDIA’s blog, or gather-and-broadcast’s ~10 TB tax remains the quiet governor on how often policies can update.
[1]