GLM-5.3 fits on two DGX Stations, and NVIDIA's own test shows the concurrency cliff

On 13 November 2026, NVIDIA published a tutorial on serving GLM-5.3 from two machines that sit on a desk, rather than from a cloud endpoint. The model is the company's NVFP4 quantization of Z.ai's mixture-of-experts design, which keeps many specialist sub-models and uses only some of them for each token. It has 753 billion parameters, with 40 billion activated per token. Sparse activation cuts the work per token. It does not shrink the memory bill, because every expert's weights still have to be on hand. NVIDIA says the checkpoint needs about 464 GB of GPU memory, more than the 252 GB of HBM3e in a single GB300 GPU.

The proposed answer is two GB300 DGX Stations, connected over ConnectX-8 RDMA, a direct memory link between the machines, with vLLM providing one OpenAI-compatible API. Fatima Fall's post is a recipe, and it is also a measurement. Startup is not the hard part. The hard part is what happens when more than a handful of requests arrive at once.

Tensor parallelism of 2 splits the model's tensors across the two GPUs. Both Stations compute every request, and Station A returns the answer. The memories of the two GPUs stay distinct. CPU weight offload is turned off, so the tutorial's premise is that all weights stay in GPU memory. Each Station also needs about 464 GB of free disk for its copy of the checkpoint, plus room for the container image. Users must accept the NVIDIA Open Model Agreement and the original GLM-5.3 license. The SPEED-Bench prompts used in the load test are under a separate NVIDIA evaluation-dataset license, and part of the preparation depends on access to the gated cais/hle dataset on Hugging Face.

The launch script sets a context ceiling of 98,304 tokens, covering input plus output, and gives vLLM 93 percent of GPU memory. Multi-token prediction is on, a form of speculative decoding in which the model's draft layer proposes four tokens per step and vLLM checks them with the full model before accepting them. FP8 storage for the key-value cache, the model's working memory of tokens already seen, halves that cache compared with 16-bit storage. Prefix caching is on, so repeated prompt prefixes are not processed again.

The load test uses AIPerf on the SPEED-Bench Coding dataset, with prefix caching enabled, four draft tokens per step, and an FP8 key-value cache. Warmup requests are excluded. Token counts include reasoning. The configured concurrencies are 1, 8, 16, and 32. At concurrency 1, overall output is 107.4 tokens per second, mean request latency is 49.84 seconds, mean time to first token is 479.75 milliseconds, and mean inter-token latency is 8.78 milliseconds. At concurrency 8, overall output rises to 386.0 tokens per second, and mean request latency is 114.17 seconds. At concurrency 16, overall output peaks at 607.8 tokens per second, with mean request latency of 136.46 seconds and mean inter-token latency of 23.77 milliseconds.

At concurrency 32 the service goes the other way. Overall output falls to 385.0 tokens per second, below the concurrency-8 figure. Mean request latency jumps to 453.54 seconds, mean time to first token to 899.36 milliseconds, and mean inter-token latency to 77.90 milliseconds. Mean per-user throughput, which is what one client actually feels, falls from 115.3 tokens per second at concurrency 1 to 13.0 at concurrency 32. For a desk-side API, that is the difference between a usable shared endpoint and a queue that takes minutes per request.

A fair reading is that NVIDIA published the cliff instead of stopping the chart at the peak. 607.8 tokens per second at 16 concurrent requests, with every weight in GPU memory, is a real local option for work that should not leave the room. The post is also explicit about the hardware bar: two validated Stations, an RDMA playbook, and a multi-hundred-gigabyte download on each box.

The numbers are still NVIDIA's, measured on NVIDIA hardware, with prefix caching and multi-token prediction turned on, on one coding dataset, counting reasoning tokens. The post says performance may change if prefix caching is turned off, and it tells readers to rerun AIPerf with their own prompts. There is no independent replication in the article, and no comparison with a cloud endpoint on the same workload. Treat Table 2 as a baseline for this recipe, not as a general speed rating for GLM-5.3.

The reason the recipe exists is not speed. NVIDIA says a local endpoint is useful for inference on sensitive or proprietary data, for prototyping agents without a per-token bill, and for work that is hard to do on a hosted service. That pitch only holds if the two-box cluster stays up, and if the team can live with the concurrency window the table describes. Past concurrency 16, the same blog that advertises desk-side frontier serving also shows multi-minute mean latencies.

The other constraint is operational. The two Stations have to be cabled, validated, and launched in the right order. The checkpoint is large enough that a 1 gigabit link would take about an hour, or about six minutes at 10 gigabits per second. This is local in the sense that the weights never go to a vendor API. It is not local in the sense of one box under a desk, turned on and forgotten.

Over the next six months, the useful check is whether later NVIDIA recipes, or independent vLLM runs, raise throughput at concurrency 32 without giving up the no-offload rule, and whether anyone publishes the same table on a non-coding workload. If the cliff stays where Table 2 puts it, two-station GLM-5.3 serving will be a low-concurrency tool for private data, not a general replacement for a hosted endpoint. The model can sit in the room. The room has to be wired for it, and the queue has to stay short.