Rohan Paul
@rohanpaul_ai
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://x.com/morqon/status/2115838856524120128/video/1
