Analysts followed an assigned lean

Anthropic published a research note on how an agent affects expert judgment. The page date is October 27, 2026. The authors are Amaç Herdağdelen, Hailey Joren, and Alex Tamkin. The page says they asked professional analysts with experience in causal inference to do a statistical analysis in Claude Code, to see whether the experts still made the decisions that matter. The task was set at a fictional farming cooperative and asked whether a voluntary, paid advisory service should be expanded. Analysts had to choose among four options that could each be defended, including whether to measure productivity as crop yield per hectare or as yield per unit of operating cost, and then give a final recommendation.

The figures come from this page, not from a replication outside it. On each decision, a coin flip set which way the agent leaned, and the analyst was not told. The agent was told to hold that lean as its own view and to make the strongest case the data allowed; if the evidence turned decisively the other way, it was to change its mind, and it was not to mislead or hide data. Across the four analytical choices together, analysts went with the agent's lean 86 percent of the time. The page says that if the agent's input made no difference, about 50 percent would be expected. Split apart, the outcome measure was 92 percent, the comparison method 82 percent, which farms the estimate covered 81 percent, and the weighting 90 percent. The final recommendation moved less. When the agent leaned toward rolling the program out, 54 percent of analysts recommended that; when it leaned the other way, 36 percent did. Using only those four leans, a statistical model correctly predicted 82 percent of the recommendations. Always guessing the most common recommendation scored 61 percent. The figure note says the four analytical decisions moved choices by 62 to 82 percentage points, and the final recommendation by 17.

People who had already stated a view also changed it. The page says that in 38 cases an analyst had expressed a preference before the agent stated its lean. When the lean went against that choice, they switched 60 percent of the time; when it agreed, only 6 percent switched. The gap is 54 percentage points. The page also says the 95 percent confidence interval runs from 24 to 82 percentage points, because there were only 38 cases, so the estimate is imprecise, and the difference cannot be pinned on the agent with certainty. A later survey asked why that choice rather than the alternative. While answering, analysts could not look back at the conversation. Even so, 77 percent repeated the agent's argument in at least one answer, sometimes word for word, and only 15 percent credited the agent anywhere. Six of the 48 guessed that the study was about the agent's influence on their choices. None guessed that the agent's own preference had been set for them.

Confidence was not clearly lower among those who followed. Analysts who followed the lean on all four analytical decisions rated confidence in the final recommendation at 71 on average; the rest rated it at 73. Fifty-two percent of analysts rated the final recommendation as more their own than the agent's. The two groups were influenced to a similar degree: the first group's choices shifted about 65 percentage points toward the agent's lean, and the second group's about 79, a difference the page places inside the margin of error.

The sample is in the design section. Sixty-one analysts were recruited. Eleven chose not to use the agent. Of the remaining 50, 65 percent held a master's or a doctorate or had six or more years of experience. Of the 48 who answered the survey, 90 percent rated themselves proficient or expert. The median time on the task was 65 minutes, and half the analysts fell between 50 and 79 minutes. The setting was a web version of VS Code with Claude Code, running Claude Opus 4.8 at High effort. With no lean assigned, the same model made the same five choices in 112 out of 123 runs. Of the 11 who worked without the agent, six made serious errors that invalidated the result. Of the 50 who used it, three did. The page says those who worked without the agent were less experienced, so the two groups cannot be compared directly.

Eighty-six percent against about 50 percent means that, in this setup, the assigned lean changed the choices. The judgment has to sit next to the limit the page states itself. The authors write that they deliberately made the agent hold and voice a preference, so the effect could be measured. The rates describe what happens when an agent leans one way and says so. The task took about an hour, the subject was farming, the analysts knew causal inference but not farming, and only one model was used. The page does not say whether the same rate would show up in real policy or research work.