Ray Train DLC: distributed training dies on the container stack, not the model code

On 28 October 2026 the AWS Machine Learning Blog published a concrete distributed-training path: use the published Ray Train Deep Learning Container (DLC) on Amazon EKS with KubeRay, and send inter-node collectives over Elastic Fabric Adapter (EFA). The opening line is the thesis—what usually stops PyTorch training from scaling out to a cluster is the container, not the model code.

Every container needs the same CUDA runtime, a PyTorch build compiled against it, NCCL, EFA userspace libraries, the AWS OFI NCCL plugin that connects NCCL to EFA, and a Ray version the cluster operator accepts. Drift in any layer becomes a project unrelated to the model. The DLC’s incremental claim is that AWS assembles, tests as one unit, publishes to Amazon ECR Public, and keeps it patched—you configure instances, GPU counts, and transport instead of rebuilding the image.

The sample pulls public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1, with a stack listed as Ray 2.58.0, PyTorch 2.13.0, CUDA 13.0.2, NCCL 2.29.7, plus EFA, the AWS OFI NCCL plugin, GDRCopy, and training libraries such as DeepSpeed, flash-attn, and Transformer Engine. One tag serves as both Ray head and workers; role comes from KubeRay rayStartParams, not from two images.

The layout is restrained: one m7i.xlarge system node runs KubeRay and a GPU-less head; two same-AZ g6.12xlarge nodes (four L4 GPUs and one EFA each) run workers. EFA traffic does not cross Availability Zones, so scripts pin GPU nodes to one AZ. Training code is a single train_func handed to Ray Train’s TorchTrainer with FSDP; batches are random token IDs. Success is defined as real FSDP collectives crossing both nodes and job completion—not a leaderboard score.

The highest-leverage mechanism warning sits in the networking row: a mismatch does not produce an error. NCCL drops to TCP sockets and the job runs to completion at a fraction of the speed. The DLC claims those libraries are built and tested together, with release gates on multi-node EFA and multi-node KubeRay training paths; the sample script inspects NCCL logs for the EFA path and exits nonzero if it is missing—usable as a CI gate.

Configuration is deliberately outside the image: GPU/EFA resource requests, FI_PROVIDER / NCCL_NET_PLUGIN / NCCL_SOCKET_IFNAME, pip packages via runtime_env, and the DLC_IMAGE tag. Requests equal limits for Guaranteed QoS so one throttled rank does not stall the other seven. These are checkable engineering choices, but still vendor self-evidence inside a sample repository. The claim that the EFA path “doubled throughput over TCP on unchanged code” needs reproduction on your instance family and message pattern.

Steelman: maintaining your own training image can lock patch cadence, private CUDA extensions, and compliance scanning; a public DLC cuts startup cost but hands part of the supply chain and version rhythm to the cloud vendor. For teams with a mature internal base image, the win is “less EFA-plugin toil,” not “first time distributed training works.”

Still unverified independently: gate-test failure rates and coverage matrix; whether train-ml-cuda-v1.1 stays “zero code change” on non-g6/L4 topologies such as p5 with GPUDirect; and false positives from the capacity-probe scripts in tight-quota regions. Fact confidence is medium-high (versions, resource names, and the silent-fallback failure mode are specific); “everyone should abandon private images” is medium.

For platform and ML-infra teams, the post reorders debugging: before another learning-rate sweep, prove collectives used EFA rather than sockets. For FinOps, the sample is candid that two GPU instances plus control plane and NAT cost about $9.50 an hour and stresses tearing down the GPU node group first—cost visibility is written into the runbook. For Ray/KubeRay users, the value is not another hello-world; it is promoting silent TCP fallback into an assertable release condition.

Over the next six months, watch whether the public Ray Train DLC becomes the default baseline for multi-node jobs on EKS, whether teams put NCCL transport assertions into CI, and whether the split—“configure outside the image, ship the stack inside”—still holds on larger models and GPUDirect instance families, or whether private images return by necessity.