Hamel Husain

Should you rely on public benchmark scores when choosing a model? @VikParuchuri argues "no" in this talk. To illustrate this, he shows specific problems found in popular OCR benchmarks: ParseBench: • A test expects 310 gigawatts of solar-panel capacity when the correct value is 303. Additionally, 73% of its chart tests ask for values that aren't printed on the chart. (4:51) • A test requires "Parish Priest:" to be bold even though those words aren't bold in the source document. (15:33) • The scorer fails to recognize heading tags placed on separate lines. (16:15) • dots.mocr ranks last among six models even though it beats the first-ranked model on content and tables. This is an example of how overall benchmark score can hide important differences in model capabilities. (14:48) olmOCR-bench: • A model correctly preserves a line break in a document and gets penalized for it. (10:30) • A test expects "Republican Party" when a document says "Republican party". Copying the capitalization correctly makes the model fail. (10:49) • A 700-line postprocessor that rewrites model output adds 1 to 2 points, more than many gaps between models. (12:12) OmniDocBench: • This benchmark uses edit distance, which penalizes a correct table written in HTML more heavily than a wrong number in Markdown. So the format matters more than the answer! (6:58) This is just the tip of the iceberg, the video contains more examples. Chapters: 0:00 Choosing a model when every vendor claims to be #1 1:54 How OCR benchmark scores improve without better OCR 5:30 Metric design and tests that reject correct answers 11:33 Postprocessing, saturation, and misleading model rankings 12:40 How to slice benchmarks by your task and check for bias 16:53 How to build fair tests with verified answers and consistent scoring 19:05 How to test OCR models on your own documents 24:10 Benchmark marketing, private evals, and how to choose a model