Elamur Research⌕
☰
home / papers / 2024 / Moral Foundations of Large Language Models

Moral Foundations of Large Language Models

Paper · 2024 · Marwa Abdulhai, Gregory Serapio-García, Clement Crepy et al.

Summary

This work examines whether widely used large language models have absorbed a particular moral and political leaning from their internet-scale training data. The authors use moral foundations theory (MFT) to frame the question. They give the 30-item Moral Foundations Questionnaire (MFQ) to PaLM and to several GPT-3 engines, including ChatGPT, Davinci 2 and 3, Curie and Babbage. Each item is posed 50 times with fresh instructions, and the majority answer is scored. The resulting profiles are compared with published human survey data grouped by self-reported political affiliation. The models are also tested under explicit political-identity prompts and across different conversational contexts. The models show distinct moral foundation profiles that stay fairly stable across contexts. Targeted prompts can push a model toward a chosen ideology or a single foundation. This induced stance also changes behavior outside the questionnaire, specifically on a dialog-based charitable donation task. The paper argues that LLMs carry implicit moral positions that can shape real applications and could be exploited, for example in politically targeted persuasion.

Key points

  • Models studied are PaLM (a 62B quantized, instruction-tuned version) and GPT-3 engines from 2.7B to 175B parameters. Ada was excluded because its answers were hard to parse, and GPT-4 was skipped because of its cost.
  • Scoring method:
    • Every MFQ item was asked 50 times, with the example ratings in the prompt randomized.
    • The model was re-instructed before each item so earlier answers would not influence later ones.
    • Foundation scores were computed from the majority-voted answers.
  • The in-depth prompting and downstream analyses focus on GPT-3 DaVinci2. It was chosen because its profile was closest to human moral foundations and because its API is publicly deployed.
  • On the donation benchmark (Wang et al., 2019), prompting the model to prioritize harm led to 39% lower donations than prompting it to prioritize loyalty.
  • Stated limitations:
    • Only one downstream task was tested.
    • The human comparison data date from 2012–2016.
    • No open-source models such as LLaMa or Mixtral were included.
    • RL safety-tuned models answer with high confidence, which reduces the sensitivity of the questionnaire.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓University of California, BerkeleyBerkeley · United States127 works · 212,523 citations🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 worksNeuroscience · 95 worksCognitive Neuroscience · 95 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Marwa Abdulhai, Gregory Serapio-García, Clement Crepy, Daria Valter, John Canny and Natasha Jaques. 2024. Moral Foundations of Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.982

@inproceedings{abdulhai2024moral,
  title = {{Moral Foundations of Large Language Models}},
  author = {Marwa Abdulhai, Gregory Serapio-García, Clement Crepy, Daria Valter, John Canny and Natasha Jaques},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.982},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.982}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 298–306, New York, NY, USA. Association for Computing Machinery.
  2. Vian Bakir. 2020. Psychological operations in digital political campaigns: Assessing cambridge analytica’s psychographic profiling and targeting. Frontiers in Communication, 5:67. find
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  4. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The values encoded in machine learning research. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 173–184. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…
  6. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157– 4… find
  7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi…
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022b. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  9. Scott Clifford and Jennifer Jerit. 2013. How words do the work of politics: Moral foundations theory and the debate over stem cell research. The Journal of Politics, 75(3):659–671. find
  10. Oliver Scott Curry, Matthew Jones Chesters, and Caspar J. Van Lissa. 2019. Mapping morality with a compass: Testing the theory of ‘morality-as-cooperation’ with a new questionnaire. Journal of Research in Personality, 78:106–124. find
  11. Robert Dale. 2021. Gpt-3: What’s it good for? Natural Language Engineering, 27(1):113–118. find
  12. Sharon L. Dawson and Graham A Tyson. 2012. Will morality or political ideology determine attitudes to climate change. find
  13. Martin V. Day, Susan T. Fiske, Emily L. Downing, and Thomas E. Trail. 2014. Shifting liberal and conservative attitudes using moral foundations theory. Personality and Social Psychology Bulletin, 40(12):1559– 1573. PMID: 25286912. find
  14. Janis L. Dickinson, Poppy McLeod, Robert Bloomfield, and Shorna Allred. 2016. Which moral foundations predict willingness to make lifestyle changes to avert climate change in the usa? PLOS ONE, 11(10):1–11. find
  15. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. find
  16. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:1–14. find
  17. Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi’s moral philosophy. arXiv preprint arXiv:2205.12771. find
  18. Leo Gao. 2021. On the sizes of openai api models. https://blog.eleuther.ai/ gpt3-model-sizes/. ↗
  19. Jesse Graham, Jonathan Haidt, and Brian Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96:1029–46. find
  20. Jesse Graham, Brian A. Nosek, Jonathan Haidt, Ravi Iyer, Spassena P. Koleva, and Peter H. Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology, 101 2:366–85. find
  21. Jonathan Haidt and Craig Joseph. 2004. Intuitive ethics: How innately prepared intuitions generate culturally variable virtues. Daedalus, 133(4):55–66. find
  22. Nick Haslam and Alan Page Fiske. 1999. Relational models theory: A confirmatory factor analysis. Personal Relationships, 6:241–250. find
  23. Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2021. Can machines learn morality? the delphi experiment. find
  24. Oliver P. John and Sanjay Srivastava. 1999. The big five trait taxonomy: History, measurement, and theoretical perspectives. find
  25. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. find
  26. Kisok Kim, Je-Sang Kang, and Seongyi Yun. 2012. Moral intuitions and political orientation: Similarities and differences between south korea and the united states. Psychological reports, 111:173–85. find
  27. Marilu Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is gpt-3? an exploration of personality, values and demographics. find
  28. Artur Nilsson, Arvid Erlandsson, and Daniel Västfjäll. 2016. The congruency between moral foundations and intentions to donate, self-reported donations, and actual donations to charity. Journal of Research in Personality, 65:22–29. find
  29. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  30. Laurie E. Paarlberg, Rebecca Nesbit, Richard M. Clerkin, and Robert K. Christensen. 2019. The politics of donations: Are red counties more donative than blue counties? Nonprofit and Voluntary Sector Quarterly, 48(2):283–308. find
  31. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2022. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251.
  32. Ashley Pilipiszyn. 2021. Gpt-3 powers the next generation of apps. https://openai.com/blog/ gpt-3-apps/. ↗
  33. Nino Scherrer, Claudia Shi, Amir Feder, and David M. Blei. 2023. Evaluating the moral beliefs encoded in llms. find
  34. Richard A. Shweder, Nancy C. Much, Manamohan Mahapatra, and Lawrence Park. 1997. The "big three" of morality (autonomy, community, divinity) and the "big three" explanations of suffering. find
  35. Gabriel Simmons. 2022. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. find
  36. Satu Annukka Vainio and Jaana-Piia Mäkiniemi. 2016. How are moral foundations associated with climate-friendly consumption? Journal of Agricultural and Environmental Ethics, 29(2):265–283. find
  37. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  38. Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good.
  39. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Ma…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdulhai-2024-moral-foundations-large-language-models/