Elamur Research⌕
☰
home / papers / 2025 / Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations

Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations

Paper · 2025 · Chaoyi Xiang, Chunhua Liu, Simon De Deyne et al.

Summary

Asking an LLM about its moral views through questionnaire items is unreliable. Its answers change with the wording of the prompt, and the questionnaires may already be in its training data. The authors take an indirect route through the model's "mental lexicon." They give Llama-3.1-8B-Instruct the same set of about 12k cue words used in the English Small World of Words study. The model's responses form an association graph (WA-L) built to match the structure of the human graph (WA-H). Words from the Moral Foundations Dictionary act as seeds and carry scores on the five Moral Foundation Theory dimensions. A random-walk procedure then spreads these scores across the whole graph, producing a Global Moral Network for humans and one for the model. The authors check the inferred scores against the crowdsourced eMFD. This global method matches human moral perception better than the earlier MAG approach, which worked on local subgraphs. The model and humans agree most on positive moral concepts and differ more on negative ones. Human associations are more concrete, emotional and varied, while the model's are more abstract and uniform. The result is a prompt-robust way to audit the values a model encodes.

Key points

  • Human data comes from the English Small World of Words (WA-H). About 90k native speakers took part, mostly from the U.S., U.K., Canada and Australia. Each cue was shown to 100 people, and each person gave up to three responses.
  • The model data (WA-L) comes from Llama-3.1-8B-Instruct, prompted with the same cues and instructions. The authors also define metrics to make sure the LLM graph is structurally comparable to the human one.
  • The seeds come from Moral Foundations Dictionary 2.0 (2041 words with hard −1/0/+1 labels per dimension). Of these, 626 words overlap with the cue set. Evaluation uses the soft-valued eMFD (scores from −1 to 1), following Ramezani and Xu (2024).
  • Scores are spread by random walks that traverse the entire association graph. It covers five dimensions: Care, Fairness, Loyalty, Authority and Sanctity. This replaces MAG's propagation within the local graph around a single cue.
  • Alignment is stronger for top positive concepts than for negative ones, with notable divergences on fairness and sanctity. The authors explain these gaps using concreteness and emotion norms. Those norms cover 98% and 85% of the top 50 cues, and the authors note that cues missing from a norms list can still be scored through their responses. Code and data are released on GitHub.

Interesting ideas

  • I1. The authors avoid asking the model directly about the property of interest. Instead they elicit a low-level behaviour (free word association) that is known to reflect that property in humans, which makes the probe less sensitive to wording and leakage.
  • I2. A small dictionary with hard labels seeds a propagation over a large graph, and the resulting soft scores are tested against a separate soft-labelled lexicon. Keeping seed and test resources distinct gives an honest check.
  • I3. Before comparing machine-generated data with human data, they first verify that the two graphs are structurally alike (same cues, same response format, diversity checks), so that observed differences are not artefacts of the setup.
  • I4. Words absent from a norms dictionary still get concreteness or emotion scores by averaging over their association neighbours. Coverage gaps are thus filled from the graph rather than left empty.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could be vetted indirectly, for example by comparing a teacher's free associations or paraphrases with human norms rather than trusting its direct answers about word senses or roles.
  • I2 (world): the compiled values and polarity layer could seed a few words per value class and spread scores over WordNet/Wikidata links, then be validated on a different lexicon held out as a control.
  • I3 (learn): silver treebanks agreed by witness parsers could be checked against gold UD distributions (label mix, attachment lengths, diversity) before being used for training or compared with gold scores.
  • I4 (world): unknown lemmas in the knowledge graph could inherit provisional attributes (concreteness, affect, value class) from linked neighbours, marked as inferred so the explanation trail stays visible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia102 works · 22,618 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 works

Cite

Cite as Chaoyi Xiang, Chunhua Liu, Simon De Deyne and Lea Frermann. 2025. Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.177

@inproceedings{xiang2025comparing,
  title = {{Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations}},
  author = {Chaoyi Xiang, Chunhua Liu, Simon {De Deyne} and Lea Frermann},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.177},
  url = {https://doi.org/10.18653/v1/2025.acl-long.177}
}

References

52 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, and Natasha Jaques. 2023. Moral foundations of large language models. arXiv preprint arXiv:2310.15337. ↗
  2. Katherine Abramski, Clara Lavorati, Giulio Rossetti, and Massimo Stella. 2024. Llm-generated word association norms. In HHAI 2024: Hybrid Human AI Systems for the Social Good, pages 3–12. IOS Press. find
  3. Guilherme F. C. F. Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo. 2024. Exploring the psychology of llms’ moral and legal reasoning. Artificial Intelligence, 333:104145. find
  4. Sotiris Anagnostidis and Jannis Bulian. 2024. How susceptible are llms to influence in prompts? arXiv preprint arXiv:2408.11865. Computer Science > Computation and Language (cs.CL). ↗
  5. M. Atari, J. Haidt, J. Graham, S. Koleva, S. T. Stevens, and M. Dehghani. 2023. Morality beyond the weird: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology, 125(5):1157–1188. find
  6. John Baldwin. 2017. Culture, prejudice, racism, and discrimination. Oxford Research Encyclopedia of Communication. Date of access 5 Oct. 2024. find
  7. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods, 46(3):904–911. find
  8. Damon Centola. 2010. The spread of behavior in an online social network experiment. Science, 329:1194– 1197. find
  9. Richard A. Charter. 1996. Note on the underrepresentation of the split-half reliability formula for unequal standard deviations. Perceptual and Motor Skills, 82(2):401–402. find
  10. Herbert H Clark. 1970. Word associations and linguistic theory. New horizons in linguistics, 1:271–286. find
  11. Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. Bias and unfairness in information retrieval systems: New challenges in the llm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24), page 11 pages, New York, NY, USA. ACM. find
  12. Simon De Deyne, Álvaro Cabana, Bing Li, Qing Cai, and Meredith McKague. 2020. A cross-linguistic study into the contribution of affective connotation in the lexico-semantic representation of concrete and abstract concepts. In CogSci. find
  13. Simon De Deyne, Yoed N. Kenett, David Anaki, and Miriam Faust. 2016. Large-scale network representations of semantics in the mental lexicon. In Michael Ramscar, Matt Jones, Melody Dye, and Ernest Klein, editors, Big Data in Cognitive Science, 1st edition, page 7. Psychology Press. find
  14. Simon De Deyne, Danielle J Navarro, Guillem Collell, and Andrew Perfors. 2021. Visual and affective multimodal models of word meaning in language and mind. Cognitive Science, 45(1):e12922. find
  15. Simon De Deyne, Danielle J. Navarro, Amy Perfors, Marc Brysbaert, and Gert Storms. 2019. The “small world of words” english word association norms for over 12,000 cue words. Behavior Research Methods, 51(3):987–1006. find
  16. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. find
  17. Rui Guo, Greg Farnan, Niall McLaughlin, and Barry Devereux. 2024. QUB-cirdan at “discharge me!”: Zero shot discharge letter generation by open-source LLM. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 664–674, Bangkok, Thailand. Association for Computational Linguistics. find
  18. Loek Halman and Ruud De Moor. 1994. Religion, churches and moral values. In The individualizing society, pages 37–65. Brill. find
  19. Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61–83. find
  20. Frederic R. Hopp, Jacob T. Fisher, Devin Cornell, Richard Huskey, and René Weber. 2021. The extended moral foundations dictionary (emfd): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behavior Research Methods, 53(1):232–246. find
  21. Yupei Du, Yuanbin Wu, and Man Lan. 2019. Exploring human gender stereotypes with word association test. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6133– 6143, Hong Kong, China. Association for Computational Linguistics. find
  22. Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024. An empirical study of llama3 quantization: From llms to mllms. arXiv preprint arXiv:2404.14047. ↗
  23. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  24. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Moralbench: Moral evaluation of llms. arXiv preprint arXiv:2406.04428. ↗
  25. Hartry H. Field. 1981. 5. Mental Representation, pages 78–114. Harvard University Press, Cambridge, MA and London, England. find
  26. Jillian Fish and Moin Syed. 2020. Racism, discrimination, and prejudice. In The Encyclopedia of Child and Adolescent Development, pages 1–12. John Wiley & Sons, Inc. find
  27. Jeremy Frimer, Jonathan Haidt, Jesse Graham, Morteza Dehghani, and Reihane Boghrati. 2017. Moral foundations dictionaries for linguistic analyses, 2.0. Unpublished Manuscript. find
  28. Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P. Wojcik, and Peter H. Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Patricia Devine and Ashby Plant, editors, Advances in Experimental Social Psychology, volume 47, pages 55–130. Academic Press. find
  29. Jesse Graham, Jonathan Haidt, and Brian A. Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5):1029–1046. find
  30. Anil Jain, Karthik Nandakumar, and Arun Ross. 2005. Score normalization in multimodal biometric systems. Pattern Recognition, 38(12):2270–2285. find
  31. Cheongwoong Kang and Jaesik Choi. 2023. Impact of co-occurrence on factual knowledge of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7721–7735, Singapore. Association for Computational Linguistics. find
  32. S. Kappal. 2019. Data normalization using median median absolute deviation mmad based z-score for robust predictions vs. min–max normalization. London Journal of Research in Science: Natural and Formal, 19(4):39–44. find
  33. Zdravka Kostova and Blagovesta Radoynovska. 2008. Word association test for studying conceptual structures of teachers and students. Bulgarian Journal of Science and Education Policy (BJSEP), 2(2):209– 231. find
  34. Chunhua Liu, Trevor Cohn, Simon De Deyne, and Lea Frermann. 2022. WAX: A new dataset for word association eXplanations. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 106–120, Online only. Association for Computational Linguistics. find
  35. Chunhua Liu, Trevor Cohn, and Lea Frermann. 2021. Commonsense knowledge in word associations and ConceptNet. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 481–495, Online. Association for Computational Linguistics. find
  36. Will Lowe. 1997. Meaning and the mental lexicon. In Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI), pages 1092– 1097. find
  37. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
  38. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In ICSE ’24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, New York, NY, USA. Association for Computing Machinery. find
  39. Aida Ramezani and Yang Xu. 2023. Knowledge of cultural moral norms in large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 428–446, Toronto, Canada. Association for Computational Linguistics.
  40. Aida Ramezani and Yang Xu. 2024. Moral association graph: A cognitive model for moral inference. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46. find
  41. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  42. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2024. Do llms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026.
  43. Ahmed Topkev. 2024. Framing Religion, pages 185– 284. Springer Nature Switzerland, Cham. find
  44. Bram Van Rensbergen, Gert Storms, and Simon De Deyne. 2015. Examining assortativity in the mental lexicon: Evidence from word associations. Psychonomic Bulletin & Review, 22:1717–1724. find
  45. David A. Walker. 2006. A comparison of the spearman-brown and flanagan-rulon formulas for split half reliability under various variance parameter conditions. Archives, 5(2). find
  46. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior Research Methods, 45(4):1191–1207. find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.
  48. Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. 2003. Learning with local and global consistency. In Advances in Neural Information Processing Systems 16 (NeurIPS 2003). MIT Press. find
  49. Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2023. Evaluating the moral beliefs encoded in llms. In Advances in Neural Information Processing Systems, volume 36, pages 51778–51809.
  50. Ji-eun Shin, Eunkook M. Suh, Kimin Eom, and Heejung S. Kim. 2018. What does “happiness” prompt in your mind? culture, word choice, and experienced happiness. Journal of Happiness Studies, 19:649– 662. find
  51. Jung-Soo Son, Vinh Bao Do, Kwang-Ok Kim, Mi Sook Cho, Thongchai Suwonsichon, and Dominique Valentin. 2014. Understanding the effect of culture on food representations using word associations: The case of “rice” and “good rice”. Food Quality and Preference, 31:38–48. find
  52. Io Taxidou and Peter M. Fischer. 2014. Online analysis of information diffusion in twitter. In Proceedings of the 23rd International Conference on World Wide Web (WWW ’14 Companion), pages 1313–1318, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiang-2025-comparing-moral-values-western-english/