CUDA 13.4 and GPU Fractioning: Kubernetes finally gets enforceable GPU memory boundaries

Many inference servers, notebooks and small fine-tunes need only a slice of a GPU. Giving each one a full device wastes capacity; sharing without hard memory caps invites noisy neighbors that allocate past their lane.

NVIDIA’s Oct 27, 2026 developer post says CUDA 13.4 closes that gap with transparent memory-boundary enforcement, and that a new open-source GPU Fractioning Operator—paired with KAI Scheduler’s NvFractions mode—makes the contract usable from Kubernetes. The claim is not that MIG is obsolete. It is that soft time-slicing and hardware partitioning are no longer the only honest options when platform teams densify multi-tenant GPUs.

CUDA already offered MIG (hardware-isolated instances), classic time-slicing (turns without memory enforcement), and MPS sm-sharing (concurrent address spaces). CUDA 13.4 adds a memory boundary that is transparent to the application: a workload can request an exact memory size, or a guaranteed minimum plus a burstable maximum when free memory exists; allocations above the limit fail as they would on a smaller GPU. Compute mode is explicit—default time-slicing, or sm-sharing—and each workload sees only its fractional memory view.

Kubernetes still needs node-side plumbing. The gpu-fractioning operator, hosted under the KAI Scheduler GitHub organization, runs next to the NVIDIA GPU Operator. It sets up and supervises MPS on every GPU node, applies each Pod container’s memory limits and compute mode before the process starts, and reports memory and compute use per fraction. Schedulers that can write the documented annotations can drive enforcement; KAI Scheduler v0.18.0 is the first integrated path via NvFractions, enabled with a Helm flag that also installs the operator as a dependency.

The primary evidence is the annotation contract and mode rules NVIDIA documents, not third-party densification benchmarks. Per-container keys include nvidia.com/container.<name>.gpu-memory.request and .limit, .gpu-compute.mode (time-slicing or sm-sharing), and .gpus.devices. Workloads may request absolute sizes (e.g. 40Gi request / 60Gi limit) or KAI portion fractions (gpu-fraction: "0.5" with optional portion limit), plus gpu-fraction-num-devices to span several GPUs with the same per-device request.

Time-slicing is serial and keeps separate error domains; unused time is redistributed. SM-sharing is parallel: SMs are allocated dynamically and settle near 1/n when all are busy, but workloads share one error domain. KAI never co-locates the two modes on one GPU, though a node may host both modes on different GPUs. Administrators enable the path with helm upgrade … --set global.nvFractions.set=true. Separately, the CUDA Toolkit 13.4 announcement describes MPS V3’s modernized control layer—scriptable CLI, named servers, TOML config, SM partition controls, and cgroup-integrated GPU memory limits—consistent with the orchestration story, though that post is a broader toolkit release note rather than a Kubernetes densification study.

The steelman against treating this as a universal densification win is MIG: hardware-isolated instances already give transparent, hard partitions without trusting an annotation-writing scheduler. Teams that need strong isolation and can afford MIG topology may gain little from software fractions—especially if their GPUs or SKUs make MIG the cleaner operational story.

The new path is also an annotation-only contract. Any scheduler that omits or mis-writes keys leaves enforcement incomplete; clusters must standardize on a writer (today KAI NvFractions is the documented first) and accept that time-slicing versus sm-sharing cannot mix on one device. SM-sharing’s collapsed error domain is a real tradeoff for untrusted multi-tenant packs. Finally, the posts ship no independent occupancy, interference, or cost-per-token numbers—only mechanism and examples. Treat densification upside as a design claim until platform teams publish their own before/after.

Who pays is the platform team packing inference replicas, interactive notebooks and small fine-tunes onto multi-tenant GPUs while defending against memory overcommit. Who benefits is the same team if fractional requests replace whole-GPU reservations without relying on applications to self-limit.

Finance and capacity planners should hear the constraints with the headline: annotation discipline, single compute mode per GPU, and sm-sharing’s shared failure domain. Without those, “fractional GPU” remains a scheduling label rather than an enforceable boundary.

Over the next six months, watch whether other Kubernetes schedulers write the same NVIDIA annotations in production, whether clusters publish densification and interference metrics under NvFractions, and whether operators default time-slicing for untrusted tenants while reserving sm-sharing for trusted packs. If those land, memory-enforced sharing becomes a standard packing tool beside MIG. If they do not, CUDA 13.4’s boundary remains real at the CUDA layer but still thin as a cluster-wide contract.