Hamel Husain

Amazing BenchmarkGate [1] reveal by @xeophon where he found serious (even comical) problems in popular benchmarks: AutomationBench: "[forces the models to] describe their request bodies in prose rather than in JSON Schema" 😅 He also fixed a number issues with the grader, which when combined with the above result in a near saturation of the benchmark! OfficeQA Pro v2: the grader uses a regex and Python-based extraction, which results in rejecting many valid answers. Additionally, there is evidence of models using web access to retrieve the answer key from HuggingFace (receipts are in the post). With these and other fixes, this again results in near saturation of the benchmark! It's very impt to look at data and not trust metrics from third parties. Better yet, you should rely on your own private evals. [1] BenchmarkGate: the gap between the [un]trustworthiness of benchmarks and the extent to which people rely on them.