TPUs have different bottlenecks from GPUs: few cores so you don’t need yo think about parallelism as much, large VMEM so easier to fuse kernels, but vector units have lower throughput so anything non matmul require much more care https://t.co/4ZdhowD46c
