要点
- Silent data corruption is a wrong but plausible result from hardware, often in a tensor-core multiply, that never raises an error and can spread into every replica's weights.
- During a 54-day Llama 3 405B pre-training run on up to 16,000 H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and only the cases that interrupted the job and were traced.
- SageMaker HyperPod already replaces nodes that report GPU errors or fail deep health checks. Those tools miss this class of fault because the hardware reports nothing.
- AWS added three checks inside the training loop: a NaN and Inf gradient check, a ring replay that recomputes another node's micro-batch and compares loss and gradient fingerprints, and a weight fingerprint after the optimizer step.
- On 4 and then 3 ml.p5.48xlarge instances, 0 alarms fired in 15,320 fault-free audit steps, a 95 percent upper bound of about 1 in 5,100. Most injected faults were caught. A weight-gradient fault audited with default kernels raised no alarm in 50 steps, and passing checkpoints were still tagged as clean.
On 11 November 2026, engineers at Amazon Web Services described a way to catch silent data corruption during training on SageMaker HyperPod. The cluster already swaps out nodes that report a GPU fault. The harder case is a GPU that returns a slightly wrong number, raises no error, and lets training continue for hours with a damaged model. Detection, the authors argue, has to come from the training job itself.
[1][2]What silent corruption is, and why usual checks miss it
Silent data corruption happens when a compute unit produces a wrong but plausible value. The usual suspect is a multiply-accumulate lane in a tensor core. There is typically no error-correcting code or parity on that arithmetic path, unlike memory. The bad number can enter activations and gradients, then every replica's weights after all-reduce, the step that averages gradients across GPUs. Loss can still look normal for a long time.
The AWS post cites Meta's Llama 3 report, listed as arXiv 2407.21783. During 54 days of 405B pre-training on up to 16,000 H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption. Those were only the interruptions that were traced to this cause. Cases that quietly weakened a checkpoint without stopping the job would not appear in that count.
HyperPod already watches for GPU errors, runs deep health checks, and replaces bad nodes. The post is explicit about the gap. ECC guards memory, not a wrong tensor-core result. Xid errors fire only when the GPU itself detects a fault. Deep health checks in the style of NVIDIA's DCGM run on an idle node and can miss defects that appear only under the job's operands, temperature, or timing.
[1][2]Three checks, and a controlled test
The authors add three checks to a PyTorch DistributedDataParallel loop. When one fires, they map the suspect rank to a node and have HyperPod replace it.
The first check looks for NaN, Inf, and overflow in each rank's gradient and in the reduced gradient, on the step they appear, before the optimizer runs.
The second is a ring replay. On audit steps, every GPU recomputes a micro-batch, a small set of training examples, from another node. It compares the loss bit for bit and the gradients by fingerprint, a short signature of the bits. This is the check aimed at finite wrong values. The post warns that a single replay can blame a healthy rank if the replaying GPU is itself faulty. Their pairing is designed to catch that pattern.
The third compares weight fingerprints after the optimizer step, and before checkpoints, so a corrupted store or a bad update is not written out as a clean snapshot.
The evidence is a controlled injection study, not a report of silent corruption caught on a frontier run. They used a SageMaker HyperPod cluster on Amazon EKS, first 4 and then 3 ml.p5.48xlarge instances, 8 NVIDIA H100 80GB GPUs each. The model was a 953-million-parameter Llama-style decoder, trained on synthetic copy-task data so any micro-batch could be rebuilt. The stack was PyTorch 2.9.0 and CUDA 13.0, with 32-bit floating-point weights and optimizer state, and 16-bit brain-float compute.
Faults were injected in the MLP down-projection of the eighth of 16 decoder blocks. On the 4-instance cluster, a one-off NaN in a forward output was caught by the first check on that step, even when it was not an audit step, and the step was not applied. Repeated finite forward faults, including a bit flip in one 16-bit element and a 1.1 times change to an output column, were named at the next audit by the second check. Weight bit flips and compensating pair changes were caught by the third check, including cases where a plain bit sum of the tensor did not change. The same alarms, steps, and ranks appeared on the 3-instance runs.
They also measured false alarms. Across six fault-free schedules, 15,320 audit steps and 3,080 weight checks produced 0 alarms. When zero events are observed, a standard statistical bound puts a 95 percent upper limit of about 1 in 5,100 on the false-alarm rate per audit step.
[1][2]The miss that belongs with the successes
One result should sit beside those catches. When a weight-gradient fault that multiplied one element by 1,000 was audited with default kernels, rather than deterministic ones, there was no alarm in 50 steps. Checkpoints at steps 20 and 40 were tagged with passing audits and weight checks, because the loss was unaffected and every copy had applied the same corrupted updates. If the corruption is consistent across replicas, agreement is not safety.
The post is plain about other limits. These checks are described for data-parallel training. The authors discuss going beyond that setting, but the numbers in the tables are for this one. The data is synthetic. The model is under a billion parameters. The faults are software-injected, not worn-out silicon under a 16,000-GPU workload. A low false-alarm bound on 150 minutes of fault-free running is useful. It is not a lifetime failure rate for a large pre-training job.
[1][2]A fair objection
A fair reading is that this is a well-instrumented engineering recipe, not a proof that silent corruption in production training is now solved. Labs that already compare checksums, rerun micro-batches, or use deterministic kernels will recognize the ideas. The contribution is packaging the three checks with HyperPod node replacement, publishing a miss case, and putting the code in an AWS Labs example. The companion repository is on GitHub under awslabs/awsome-distributed-ai, in examples/use-cases/pytorch-ddp-sdc-detection.
[1][2]What to watch in the next six months
The test that would move a skeptic is not another injected-fault table on a 953-million-parameter copy task. It is a public miss-and-catch log from a real multi-week run, with audit frequency, kernel mode, and the cost of ring replay stated in wasted steps or extra GPU-hours. If default-kernel audits still miss consistent backward-path faults, teams will need deterministic kernels on audit steps, or a further check that does not depend on disagreement between replicas. Watch whether those settings become the default in HyperPod training recipes, or remain a blog companion repository.
[1][2]