98% utilization did not mean the GPU was full
A Databricks engineering post on October 14 says they sped up fine-tuning of Qwen 3.8 27B by about 6x. The site RSS time is 11:30 UTC, 19:30 in Beijing. The baseline took 7 hours 44 minutes on the full dataset, with 98% average GPU utilization and 9,396 tokens per second.
The post's point is that utilization only says the GPU was doing something, not that it was doing useful work the whole time. They saw the CPU taking longer to set up a step than the GPU took to run it, with hundreds of thousands of small operations, so the CPU could not enqueue work fast enough.
[1]
First they swapped in existing fused kernels
The loop uses transformers and the supervised-fine-tuning set HuggingFaceH4/ultrachat_200k. FSDP2 shards weights and optimizer state across GPUs, and activation checkpointing trades compute for memory so the batch per GPU can be larger. Batches are two-dimensional, at most 8,192 tokens, and short examples are padded to the longest row in the batch.
Default attention uses torch SDPA. The post says many of the small operations come from the nonstandard gated-delta layers. After installing flash-linear-attention and causal-conv1d, end-to-end time fell to 2 hours 5 minutes, about 3.7x. Operations in a profiled step fell by about 15x. The post also says an inference-only kernel is not enough: training needs a forward and a backward.
[1]Then they stopped padding the empty space
A rectangular batch pads short examples. flash-linear-attention does not take nested tensors, so they packed examples into one dimension and passed each example's position ids, so a token attends only inside its own example. Time fell from 2 hours 5 minutes to 1 hour 17 minutes.
The main reason is more sequences per batch and about 36% fewer batches. How much that saves depends on how long the examples are. The secondary reason is that the fused kernels had been compiling for each changing shape; packing made the shape stable and removed a few minutes of compilation. Together, that is the roughly 6x in the post.
[1]The sixfold figure is this fine-tune
The post says this change makes the input pipeline more complex, and there is a high chance of introducing errors that affect model quality. They judged the 36% time saving worth that risk here. The post does not include a loss or evaluation table showing whether quality moved.
It also does not say the same multiple will show up on another model, another dataset, or pretraining. Utilization was already 98% and the run still got faster, so that 98% was not evidence that the slack was gone.
[1]