Elamur Research⌕
☰
home / papers / 2026 / Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

Paper · 2026 · Jingting Zheng, Yuqi Ren, Linhao Yu et al.

Summary

Most moral evaluations of LLMs score answers one item at a time. They do not show how a model ranks competing values against each other as a whole system. This paper addresses that gap by adapting Q methodology so that humans and models complete the same task. The authors built a 140-item English Q-set based on Schwartz's value circumplex, with Moral Foundations Theory and Morality-as-Cooperation tags to describe each item's content. Both groups sort the statements into a fixed nine-level distribution. LLMs produce a strict full ranking, which is then converted to buckets by a deterministic rule. A three-factor structure derived from 35 human participants serves as the reference. Twelve models from four families are compared against it using Procrustes similarity and RSA-based Spearman correlation, along with region-wise breakdowns. Model families differ substantially, and some models are sensitive to temperature. Good global scores can also mask local distortions in particular value regions. The work matters because it adds a structural diagnostic layer to standard itemwise moral benchmarks.

Key points

  • The instrument has 140 statements. Each has one primary Schwartz value label (from the 19 refined values) plus optional MFT/MAC tags, and two experts checked coverage. Items are sorted from −4 to +4 under fixed quotas.
  • The human reference (N = 35) yields a k = 3 factor geometry. The authors treat it as specific to this instrument and sample, not as a universal morality.
  • The study covers 12 LLMs and 240 replicated Q-sorts at two temperature settings. Alignment is measured with Procrustes ϕ and Spearman ρ, and regions with fewer than 8 items are masked.
  • Rank-based and bucket-based analyses agree closely, but prompt wording matters. For GPT-5.1, mean-profile correlations across paraphrases were only 0.099 at T = 0 and 0.302 at T = 0.7.
  • Some runs collapse into low-rank replicate-by-item matrices, so no geometry can be estimated. In those cases, ϕ and ρ are reported as undefined rather than as low alignment. Brief free-text rationales were also collected for the 16 extreme placements.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tianjin UniversityTianjin · China41 works · 1,128 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksSafety Research · 543 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Jingting Zheng, Yuqi Ren, Linhao Yu, Yongqi Leng and Deyi Xiong. 2026. Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.830

@inproceedings{zheng2026beyond,
  title = {{Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts}},
  author = {Jingting Zheng, Yuqi Ren, Linhao Yu, Yongqi Leng and Deyi Xiong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.830},
  url = {https://doi.org/10.18653/v1/2026.acl-long.830}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marwa Abdulhai, Gregory Serapio-García, Clement Crepy, Daria Valter, John Canny, and Natasha Jaques. 2024. Moral foundations of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17737–17752, Miami, Florida, USA. Association for Co…
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. Preprint, arXiv:1606.06565.
  3. Radoslav Baltezarević and Ivana Baltezarević. 2024. Benefits and risks of ChatGPT in future education. Megatrend Revija, 21:71–84. find
  4. Pablo Biedma, Xiaoyuan Yi, Linus Huang, Maosong Sun, and Xing Xie. 2024. Beyond human norms: Unveiling unique values of large language models through interdisciplinary approaches. Preprint, arXiv:2404.12744. — [arXiv:2404.12744](https://arxiv.org/abs/2404.12744) find
  5. Raymond B. Cattell. 1966. The scree test for the number of factors. Multivariate Behavioral Research, 1(2):245–276. find
  6. Oliver Scott Curry, Matthew Jones Chesters, and Caspar J. Van Lissa. 2019. Mapping morality with a compass: Testing the theory of ‘morality-as-cooperation’ with a new questionnaire. Journal of Research in Personality, 78:106–124. find
  7. Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, and 1 others. 2022. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Resear… find
  8. Joshua R. de Leeuw. 2015. jsPsych: A JavaScript library for creating behavioral experiments in a web browser. Behavior Research Methods, 47(1):1–12. find
  9. Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York. find
  10. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computatio… find
  11. J. C. Gower. 1975. Generalized procrustes analysis. Psychometrika, 40(1):33–51. find
  12. Jesse Graham, Jonathan Haidt, and Brian A. Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5):1029–1046. find
  13. Jesse Graham, Brian A. Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H. Ditto. 2011. Mapping the moral domain. Journal of Personality and Social Psychology, 101(2):366–385. find
  14. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. 2023. Evaluating Large Language Models: A comprehensive survey. Preprint, arXiv:2310.19736.
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning AI with shared human values. Preprint, arXiv:2008.02275.
  16. John L. Horn. 1965. A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2):179–185. find
  17. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2025. Moral-Bench: Moral evaluation of LLMs. SIGKDD Explorations Newsletter, 27(1):62–71. find
  18. Henry F. Kaiser. 1958. The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3):187–200. find
  19. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, and 4 other… find
  20. Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. 2008. Representational similarity analysis— connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4. find
  21. Jiaang Li, Antonia Karamolegkou, Yova Kementchedjhieva, Mostafa Abdou, Sune Lehmann, and Anders Søgaard. 2023. Structural similarities between language models and neural response measurements. Preprint, arXiv:2306.01930. — [arXiv:2306.01930](https://arxiv.org/abs/2306.01930) find
  22. Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. 2025. Benchmarking distributional alignment of Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long P…
  23. Allen Nie, Yuhui Zhang, Atharva Amdekar, Chris Piech, Tatsunori Hashimoto, and Tobias Gerstenberg. 2023. MoCa: Measuring human-language model alignment on causal and moral judgment tasks. Preprint, arXiv:2310.19677.
  24. Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. ValueBench: Towards comprehensively evaluating value orientations and understanding of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  25. Peter H. Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10. find
  26. Shalom H. Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Mark P. Zanna, editor, Advances in Experimental Social Psychology, volume 25, pages 1–65. Academic Press. find
  27. Shalom H. Schwartz. 2012. An overview of the Schwartz theory of basic values. Online Readings in Psychology and Culture, 2(1). find
  28. Shalom H. Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kürşad Demirutku, Ozlem Dirilen-Gumus, and Mark Konty. 2012. Refining the theory of basic individual values. Journal of Personality and Social… find
  29. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. 2022. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. Preprint, arXiv:2210.01790. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) find
  30. Hua Shen, Nicholas Clark, and Tanu Mitra. 2025. Mind the value-action gap: Do LLMs act in alignment with their values? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 3097–3118, Suzhou, China. Association for Computational Linguistics.
  31. Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6859–6877, Albuquerque, New Mexico. Association for Computational Linguistics. find
  32. Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large Language Model alignment: A survey. Preprint, arXiv:2309.15025.
  33. Yang Yang, Luan Li, Simon de Deyne, Bing Li, Jing Wang, and Qing Cai. 2024. Unraveling lexical semantics in the brain: Comparing internal, external, and hybrid language models. Human Brain Mapping, 45(1):e26546. find
  34. Ridwan Islam Sifat. 2025. Commentary—AI and public policy: Navigating the possibilities and limitations. Politics & Policy, 53(1):e70019. find
  35. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others… find
  36. Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwang Song, Tingting Cui, Xiaoqing Cheng, Tao Liu, and Deyi Xiong. 2024. CMoralEval: A moral evaluation benchmark for Chinese Large Language Models. In Findings of the Association for Computational Linguistics: ACL… find
  37. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L. Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. Position: A roadmap to pluralistic alignment. In Proceedings of the 41st International Conference o…
  38. William Stephenson. 1953. The Study of Behavior: Q-Technique and Its Methodology. University of Chicago Press, Chicago. find
  39. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Preprint, arXiv:2305.04388.
  40. Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, David Stillwell, Luning Sun, Fang Luo, and Xing Xie. 2023. Evaluating general-purpose AI with psychometrics. Preprint, arXiv:2310.16379. — [arXiv:2310.16379](https://arxiv.org/abs/2310.16379) find
  41. Simon Watts and Paul Stenner. 2012. Doing Q Methodological Research: Theory, Method and Interpretation. SAGE Publications, London. find
  42. Shaoyang Xu, Weilong Dong, Zishan Guo, Xinwei Wu, and Deyi Xiong. 2024. Exploring multilingual concepts of human values in Large Language Models: Is value alignment consistent, transferable and controllable across languages? In Findings of the Association for Computational Linguistics: EMNLP 2024, p… find
  43. Shaoyang Xu, Yongqi Leng, Linhao Yu, and Deyi Xiong. 2025. Self-pluralising culture alignment for Large find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-beyond-value-benchmarks-measuring-value/