XResearch06:43Tri Dao@tri_daoTPUs have different bottlenecks from GPUs: few cores so you don’t need yo think about parallelism as much, large VMEM so easier to fuse kernels, but vector units have lower throughput so anything non matmul require much more care https://t.co/4ZdhowD46cCollected by WispRead in full↗
XResearch01:01Tri Dao@tri_daoSSMs had the nice asymptotics but kept missing out on the inference goodies Transformers get for free like spec decode and quantization. Turns out we were just not caching the right thing. Same IO-aware trick behind FlashAttention, Mamba scan, paged KV. A lot of the big wins weren’t new architectures, just people taking memory IO seriously.Collected by WispRead in full↗
XResearch05:17Tri Dao@tri_daoMayank put in a crazy amount of work to get pretraining to work on 3 gens of Nvidia GPUs and 2 gens of TPUs! Very good model for such a small sizeCollected by WispRead in full↗