60.3 percent on a clean call, 34.0 on all three

The visible date on this Sierra post is October 22, 2026. The benchmark it introduces, τ-Entity, measures whether a voice agent stores one field exactly. A related post listed at the bottom, τ-i18n, is dated October 21, 2026, and it measures something else. The figures on this page are not carried over from that one. The benchmark has 200 tasks and ten entity types, including names, phone numbers, addresses, and emails. Each call focuses on a single field. Success requires the stored value to exactly match the expected value. The caller cooperates and spells accurately when asked. Most calls take less than two minutes.

The leaderboard pools six languages. GPT Live 1 leads: it captures the value correctly in 60.3 percent of regular calls, but succeeds under all three conditions, regular, noise-heavy, and speech-heavy, on only 34.0 percent of tasks. Gemini 3.8 Live reaches 43.8 percent in regular conditions and 19.5 percent across all three. Grok Voice Think Fast 2.0 reaches 44.2 percent and 24.0 percent. English alone is easier. GPT Live 1 reaches 74.0 percent on regular calls and 47.5 percent across all three. Spanish, Portuguese, and Hindi stay close to English, and for Gemini and Grok they score above English. Korean falls to 3.5 to 8.5 percent across all three conditions, and Mandarin to 12.5 to 24.5 percent. Those are ranges across agents, not one model's single score.

The same page also reports a comparison from the paper. It is not another column on the leaderboard. When agents choose their own verification strategy, only 14 to 41 percent of tasks succeed across all three conditions. The prescribed sequence is: ask, have the caller spell the value or read it back, confirm and correct, then submit. That sequence raises success to 37 to 54 percent, a gain of 14 to 31 percentage points, and the calls take 21 to 28 seconds longer. Each configuration runs the same 200 tasks in each of three conditions under both prompts: 4,800 calls in total. This story records that sequence as a condition the paper tested, and records the change in success and time. It does not turn the sequence into setup instructions.

Realizing a capture may be wrong does not guarantee a repair. In the paper's behavioral analysis, only 24 to 37 percent of initially incorrect captures that agents verify are ultimately corrected. Sometimes the agent repeats the same mistake. Sometimes it replaces the value with a new incorrect one. Errors accumulate when a call needs several fields. In the paper's multi-field study, only 53 percent of three-field calls succeed even though individual fields remain 75 percent accurate. On those same multi-field tasks, validating each field and allowing retries increased success from 64 percent to 82 percent. The page says the intervention combines validation and retry, so the two cannot be separated. The 200 tasks on the τ-Entity leaderboard each focus on a single field. The 53 percent is not that leaderboard's pass rate.