Hamel Husain

There is a false equivalence being made between open source software and AI benchmarks. The argument is that we shouldn't expend effort criticizing benchmarks, and should instead improve them. Unlike OSS software, benchmark results are consumed for decision making at face value. They are currently used like Yelp or a credit score, and less like a informative dataset you can analyze Benchmarks don't come with disclaimers, and people don't realize how brittle they can be. Consumers don't have the data skills to analyze the traces and see the limitations of benchmarks. Hell people who have the skills don't question it either, and are SHOCKED when they see the traces. So yeah, I think more people should have healthy skepticism. Right now, most people don't realize what they are looking at. The biggest problem is the large dissonance between the marketing of benchmarks and what they really measure. Instead of deflecting the criticisms of benchmarks, I think the communications need to be fixed around them.