Jason Wei

Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: https://t.co/GswjxqWz1c Some observations: - HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption - Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark) - Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor - Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔 - In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA) - DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it - Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year