Elamur Research⌕
☰
home / papers / 2023 / Evaluating the Moral Beliefs Encoded in LLMs

Evaluating the Moral Beliefs Encoded in LLMs

Paper · 2023 · Nino Scherrer, Claudia Shi, Amir Feder et al.

Summary

The paper examines which moral preferences large language models reveal when treated as survey respondents. It also addresses two practical obstacles: models emit token strings rather than discrete choices, and their answers shift with prompt wording.

The authors propose a statistical framework with these components:

  • Action likelihood groups semantically equivalent outputs into probabilities over actions.
  • Marginal action likelihood averages those probabilities across several question formats.
  • Entropy-based measures capture the model's uncertainty.
  • Two consistency diagnostics assess sensitivity to wording: QF-C, based on the Generalized Jensen-Shannon divergence, and QF-E.

They apply this framework to MoralChoice, a new dataset of hypothetical dilemmas with two candidate actions each, and survey 28 open- and closed-source LLMs.

When the right answer is obvious, most models pick the commonsense option. On genuinely hard dilemmas, most models hedge. Exceptions exist: some models waver on easy cases, often because of prompt sensitivity, and several commercial models hold firm and highly similar preferences on hard dilemmas. These results argue for testing models with multiple prompt variants and raise questions about how preference fine-tuning shapes moral behaviour.

Key points

  • MoralChoice contains 687 low-ambiguity and 680 high-ambiguity scenarios. Each scenario is labelled with the rules it violates, such as "do not kill." The data are on Hugging Face (ninoscherrer/moralchoice), and the code is on GitHub (moralchoice).
  • Question forms: each scenario is posed in six versions, built from three templates (A/B, Repeat, Compare) crossed with two action orderings. Responses are sampled at temperature 1.
  • Commercial sub-cluster: gpt-4, Anthropic's Claude models and Google's text-bison-001 (PaLM 2) agree strongly with one another, with pairwise correlations of at least 0.75. Their correlation with the cluster of open-source and AI21 models is at most 0.28.
  • Source inconsistency: the Claude model in this sub-cluster is named claude-v1.3 in one place and claude-v1.1 in another, alongside claude-instant-v1.1.
  • Stated limitations:
    • The scenarios are built around norm violations and lack diversity.
    • Only three English prompt templates are used.
    • Each question is asked with a freshly reset context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Nino Scherrer, Claudia Shi, Amir Feder and David M. Blei. 2023. Evaluating the Moral Beliefs Encoded in LLMs. arXiv:2307.14324.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.14324 · arXiv 2307.14324

@misc{scherrer2023evaluating,
  title = {{Evaluating the Moral Beliefs Encoded in LLMs}},
  author = {Nino Scherrer, Claudia Shi, Amir Feder and David M. Blei},
  year = {2023},
  doi = {10.48550/arxiv.2307.14324},
  eprint = {2307.14324},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.14324}
}

References

62 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abdulhai, S. Levine, and N. Jaques. “Moral Foundations of Large Language Models”. In: AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI (2022).
  2. Gati Aher, Rosa I Arriaga, and Adam Tauman Kalai. “Using Large Language Models to Simulate Multiple Humans”. In: arXiv:2208.10264 (2022). find
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety”. In: arXiv:1606.06565 (2016). Jacob Andreas. “Language Models as Agent Models”. In: Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.
  4. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. “PaLM 2 Technical Report”. In: arXiv:2305.10403 (2023). Anthropic. API Reference Documentation. 2023.
  5. Karl Aquino and Americus Reed II. “The Self-Importance of Moral Identity.” In: Journal of Personality and Social Psychology 6 (2002). find
  6. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. “Out of One, Many: Using Language Models to Simulate Human Samples”. In: arXiv:2209.06899 (2022).
  7. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. “A General Language Assistant as a Laboratory for Alignment”. In: arXiv:2112.00861 (2021). find
  8. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”. In: arXiv:2204.05862 (2022). find
  9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. “Constitutional AI: Harmlessness from AI Feedback”. In: arXiv:2212.08073 (2022).
  10. Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, and Christopher Summerfield. “Fine-Tuning Language Models to Find Agreement among Humans with Diverse Preferences”. In: Neural Inform… find
  11. Ziv Bar-Joseph, David K Gifford, and Tommi S Jaakkola. “Fast Optimal Leaf Ordering for Hierarchical Clustering”. In: Bioinformatics suppl_1 (2001). find
  12. Emily M Bender and Alexander Koller. “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data”. In: Annual Meeting of the Association for Computational Linguistics. 2020.
  13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  14. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. “Sparks of Artificial General Intelligence: Early Experiments with GPT-4”. In: arXiv:2303.12712 (2023).
  15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. “PaLM: Scaling Language Modeling with Pathways”. In: arXiv:2204.02311 (2022).
  16. Julia F Christensen, Albert Flexas, Margareta Calabrese, Nadine K Gut, and Antoni Gomila. “Moral Judgment Reloaded: A Moral Dilemma Validation Study”. In: Frontiers in psychology (2014). find
  17. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, A… find
  18. Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. “Inducing Anxiety in Large Language Models Increases Exploration and Bias”. In: arXiv:2304.11111 (2023). Cohere. Cohere Command Documentation. 2023. — [arXiv:2304.11111](https://arxiv.org/abs/2304.11111) find
  19. Avia Efrat and Omer Levy. “The Turking Test: Can Language Models Understand Instructions?” In: arXiv:2010.11982 (2020).
  20. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. “Measuring and Improving Consistency in Pretrained Language Models”. In: Transactions of the Association for Computational Linguistics (2021). find
  21. Naomi Ellemers, Jojanneke Van Der Toorn, Yavor Paunov, and Thed Van Leeuwen. “The Psychology of Morality: A Review and Analysis of Empirical Studies Published From 1940 Through 2017”. In: Personality and Social Psychology Review 4 (2019). find
  22. Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. “Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences”. In: Conference on Empirical Methods in Natural Language Processing. 2021. find
  23. Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. “Social Chemistry 101: Learning to Reason about Social and Moral Norms”. In: arXiv:2011.00620 (2020). find
  24. Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. “Does Moral Code Have a Moral Code? Probing Delphi’s Moral Philosophy”. In: arXiv:2205.12771 (2022). find
  25. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. “The Capacity for Moral Self-Correction in Large Language Models”. In: arXiv:2302.07459 (2023).
  26. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”. In: arXiv:2209.07858 (2022). Bernard Gert. Common M…
  27. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. “Improving Alignment of Dialogue Agents via Targeted Human Judgements”. In: arXiv:2209.14375 (2022).
  28. Jesse Graham, Jonathan Haidt, and Brian A Nosek. “Liberals and Conservatives Rely on Different Sets of Moral Foundations.” In: Journal of Personality and Social Psychology 5 (2009). find
  29. Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. “Mapping the Moral Domain.” In: Journal of Personality and Social Psychology 2 (2011). find
  30. Joshua D Greene, Fiery A Cushman, Lisa E Stewart, Kelly Lowenberg, Leigh E Nystrom, and Jonathan D Cohen. “Pushing Moral Buttons: The Interaction between Personal Force and Intention in Moral Judgment”. In: Cognition 3 (2009). find
  31. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. “The Political Ideology of Conversational AI: Converging Evidence on ChatGPT’s Pro-Environmental, Left-Libertarian Orientation”. In: arXiv:2301.01768 (2023).
  32. Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. “Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs”. In: arXiv:2111.13654 (2021). find
  33. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. “Aligning AI With Shared Human Values”. In: International Conference on Learning Representations (2021).
  34. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. “Unsolved Problems in ML Safety”. In: arXiv:2109.13916 (2021).
  35. John J Horton. “Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?” In: arXiv:2301.07543 (2023). — [arXiv:2301.07543](https://arxiv.org/abs/2301.07543) find
  36. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. “OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization”. In: arXiv:2212.12017 (2022).
  37. Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. “BECEL: Benchmark for Consistency Evaluation of Language Models”. In: International Conference on Computational Linguistics. 2022. find
  38. Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. “Can Machines Learn Morality? The Delphi Experiment”. In: arXiv:2110.075…
  39. Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. “When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment”. In: Neural Information Processing Systems (2022). find
  40. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. “Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation”. In: International Conference on Learning Representations. 2023. AI21 Labs. Jurassic-2 Models Documentation. 2023.
  41. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. “The Flan Collection: Designing Data and Methods for Effective Instruction Tuning”. In: arXiv:2301.13688 (2023).
  42. Nicholas Lourie, Ronan Le Bras, and Yejin Choi. “Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes”. In: AAAI Conference on Artificial Intelligence. 2021. David JC MacKay. Information Theory, Inference and Learning Algorithms. 2003. find
  43. I Dan Melamed. “Measuring Semantic Entropy”. In: Tagging Text with Lexical Semantics: Why, What, and How? 1997. find
  44. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. “Crosslingual Generalization through Multitask Finetuning”. In: arXiv:2211.01786 (2022). Daniel Müllner. “Modern Hierarchical, Agglome…
  45. Allen Nie, Yuhui Zhang, Atharva Amdekar, Christopher J Piech, Tatsunori Hashimoto, and Tobias Gerstenberg. MoCa: Cognitive Scaffolding for Language Models in Causal and Moral Judgment Tasks. 2023. OpenAI. GPT-4 Technical Report. 2023.
  46. OpenAI. Models Documentation. 2023. find
  47. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. “Training Language Models to Follow Instructions with Human Feedback”. In: Neural Information Processing Systems (2022).
  48. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. “Generative Agents: Interactive Simulacra of Human Behavior”. In: arXiv:2304.03442 (2023). find
  49. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. “Social Simulacra: Creating Populated Prototypes for Social Computing Systems”. In: Annual ACM Symposium on User Interface Software and Technology. 2022.
  50. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain,… — Discovering Language Model Behaviors with Model-Written Evaluations
  51. Steven T Piantasodi and Felix Hill. “Meaning Without Reference in Large Language Models”. In: arXiv:2208.02957 (2022). — [arXiv:2208.02957](https://arxiv.org/abs/2208.02957) find
  52. James R Rest. “Longitudinal Study of the Defining Issues Test of Moral Judgment: A Strategy for Analyzing Developmental Change.” In: Developmental Psychology 6 (1975). find
  53. Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. “Are Red Roses Red? Evaluating Consistency of Question-Answering Models”. In: Annual Meeting of the Association for Computational Linguistics. 2019. find
  54. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. “Whose Opinions Do Language Models Reflect?” In: arXiv:2303.17548 (2023). Murray Shanahan. “Talking About Large Language Models”. In: arXiv:2212.03551 (2022).
  55. Richard A Shweder, Nancy C Much, Manamohan Mahapatra, and Lawrence Park. “The “Big Three” of Morality (Autonomy, Community, Divinity) and the “Big Three” Explanations of Suffering”. In: Morality and health. 2013. Robin Sibson. “Information Radius”. In: Zeitschrift für Wahrscheinlichkeitstheorie und… find
  56. Gabriel Simmons. “Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity”. In: arXiv:2209.12106 (2022). find
  57. Irene Solaiman and Christy Dennison. “Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets”. In: Neural Information Processing Systems (2021).
  58. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. “Learning to Summarize with Human Feedback”. In: Neural Information Processing Systems (2020). find
  59. Albert Webson and Ellie Pavlick. “Do Prompt-Based Models Really Understand the Meaning of Their Prompts?” In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
  60. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. “HuggingFace’s Transformers: State-of-the-art Natural Language Processing”. In: arXiv:1910.03771 (2019).
  61. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. “Calibrate Before Use: Improving Few-Shot Performance of Language Models”. In: International Conference on Machine Learning. PMLR. 2021.
  62. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. “Fine-Tuning Language Models from Human Preferences”. In: arXiv:1909.08593 (2019). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/scherrer-2023-evaluating-moral-beliefs-encoded-llms/