Rohan Paul

Absolutely beautiful new paper from Stanford Univ + Together AI "Intelligence per Watt: Measuring Intelligence Efficiency of Local AI" - It found that hybrid local-cloud routing reduced energy, compute, and cost by 60% to 80% against its batched-cloud baseline. - From 2023 to 2025, local AI’s intelligence-per-watt improved 5.3×, while locally serviceable query coverage jumped from 23.2% to 71.3%. - An iPhone 16 Pro achieved roughly 7× higher intelligence-per-watt than workstation GPUs on the same model and precision, showing how power-efficient mobile AI can be for lightweight queries. - A diverse pool of 20+ local models actually beat the paper’s 3 frontier cloud models on 3 of 4 benchmarks when each query was routed to the best model, showing how much model diversity can matter. - Dropping precision from FP16 to FP4 cut inference energy by 3X–3.5X, while costing roughly 2.5 percentage points of accuracy per precision step; in one test, a larger FP4 model even beat a smaller FP16 model. - The biggest remaining weakness is concentrated at the hard end: on the paper’s hardest reasoning slice, about 95% of problems were still unsolved by local models, even as easier and medium-difficulty tasks improved rapidly.