Rohan Paul

New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help. Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel. They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request. The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent. Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.