Rohan Paul

This paper finds that LLM moral choices often lack a stable ranking of values, and averaging their answers creates a fake, human-like moral hierarchy. Many studies conclude that LLMs hold sensible moral values because they do well on morality tests. But good answers do not show that a model weighs competing values against each other. The researchers gave 5 LLMs, including GPT-5.2, Gemini, and DeepSeek, pairs of moral violations and asked which was worse. They read each answer's probability straight from the model's next-token output, then tested whether the choices fit a consistent ranking. Reversing the order of the 2 options changed the ranking in 26 of 30 test blocks. After averaging, 4 of 5 models showed a clean, nearly identical ranking that matched no individual test set. Before trusting a model's moral alignment scores, check that its choices hold a consistent ranking on individual prompts and both answer orders, without averaging.