The problem
LLM assistants are widely used for everyday social advice — how to reply to a colleague, how to decline politely. Evaluating that ability has been hard for two reasons: the setup requires an assistant to understand a social situation from a user's subjective narrative, and social properties such as another person's intentions usually have no verifiable ground truth.
The method
In a paper posted to arXiv on September 15, "Verifiable Social Reasoning for LLM Assistants" (arXiv:2609.17496), Google researchers introduce Fuse, a multi-agent simulation framework. A target agent with a hidden motive interacts with other agents, including one representing the user; the user then consults the assistant being evaluated and asks it to infer the target's motive. Because the motive is constructed, correctness is checkable — ground truth is guaranteed by construction.
The author list includes Google vice presidents Marian Croak and Yossi Matias. The team validated simulation faithfulness with 24,000 human annotations and applied Fuse to 12 LLMs.
Findings
Four results are worth isolating:
- user mediation compounds the inherent difficulty of social reasoning;
- LLMs show systematic sensitivity to biased user framing — how the story is told matters more than what happened;
- models sometimes need more detail than humans to reach a correct prediction;
- longer conversations do not always improve performance, even with more chances to ask clarifying questions.
The team open-sourced Fuse and a dataset of 21,000 examples.
The caveat
This is an arXiv preprint, not peer-reviewed. The construction — a hidden motive set by the system — is still a distance from real social settings, where users often cannot articulate their own goals. But "constructed ground truth" gives social-reasoning evaluation a reproducible yardstick for the first time.
[1]