Elamur Research⌕
☰
home / papers / 2025 / Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework

Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework

Paper · 2025 · Mohna Chakraborty, Lu Wang and David Jurgens

Summary

Large language models often reach moral verdicts by relying on superficial textual cues, which leads to biased or inconsistent decisions in ethically complex situations. The authors propose a value-grounded framework for evaluating and improving moral reasoning. Each item is treated as a binary choice, such as support versus oppose, paired with a free-text justification. Prompts come from a taxonomy with three families:

  • Value systems: Moral Foundations Theory, Schwartz's Value Theory, Hofstede's Cultural Dimensions.
  • Ethical theories: Care Ethics, Contractarianism, Deontology, Ethical Pluralism, utilitarianism.
  • Cognitive strategies: First-Principles, Step-by-Step, Consequentialist, Counterfactual reasoning.

Twelve open-source models were tested on four moral benchmarks against two baselines: label-only answers and reasoning without guidance. Explicitly structured prompts consistently raised accuracy and coherence. First-Principles reasoning and the Schwartz + Care Ethics combination worked best, and smaller models benefited most. Supervised distillation then transferred structured justifications from large teacher models into a 3B student, closing much of the gap with no added inference cost. This offers an inexpensive route toward more interpretable and value-aware language systems.

Key points

  • The benchmark covers 12 open-source LLMs on four datasets: Value Kaleidoscope, UniMoral, MoralCoT and ETHICS.
  • The work is organized around four research questions:
    • whether reasoning beats direct prediction;
    • which value or ethical frameworks help most;
    • which cognitive strategies help most;
    • whether moral competence can be distilled into small models.
  • First-Principles Reasoning was the most robust strategy across architectures and datasets, followed closely by the Schwartz + Care Ethics scaffold.
  • For distillation, Mistral Large Instruct (2407) and LLaMA-3.3 Instruct (70B) served as the teachers, while LLaMA-3.2 (3B) was the student. Accuracy improved on three datasets, with the largest gains under First-Principles prompting.
  • The authors list several limitations:
    • the set of frameworks is mostly Western and not exhaustive;
    • the datasets are curated;
    • students inherit their teachers' biases;
    • evaluation covers only single-turn settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksSafety Research · 543 worksDecision Sciences · 209 worksNeuroscience · 165 worksCognitive Neuroscience · 164 worksInformation Systems and Management · 41 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Mohna Chakraborty, Lu Wang and David Jurgens. 2025. Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1541

@inproceedings{chakraborty2025structured,
  title = {{Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework}},
  author = {Mohna Chakraborty, Lu Wang and David Jurgens},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1541},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1541}
}

References

62 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. Phi-4 technical report. Larry Alexander and Deontological ethics. Michael Moore. 2007. find
  2. Maryam Amirizaniani, Elias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. 2024. Can llms reason like humans? assessing theory of mind reasoning in llms for open-ended questions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 34–44. find
  3. Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018. The moral machine experiment. Nature, 563(7729):59–64. find
  4. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and 1 others. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on A… find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  7. Fiery Cushman. 2013. Action, outcome, and value: A dual-system framework for morality. Personality and social psychology review, 17(3):273–292. find
  8. John Dewey and James Hayden Tufts. 2022. Ethics. DigiCat. find
  9. Ronald Dworkin. 2013. Taking rights seriously. A&C Black. pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55–130. Elsevier. find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  11. Joshua D Greene, R Brian Sommerville, Leigh E Nystrom, John M Darley, and Jonathan D Cohen. 2001. An fmri investigation of emotional engagement in moral judgment. Science, 293(5537):2105–2108. find
  12. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024. Olmo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui… find
  13. Katharina Haemmerl, Bjoern Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin Rothkopf, Alexander Fraser, and Kristian Kersting. 2023. Speaking multiple languages affects the moral bias of language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 21… find
  14. Denis Emelin, Ronan Le Bras, Jena D Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718. find
  15. Jonathan Haidt. 2001. The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological review, 108(4):814. find
  16. Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4151–4171.
  17. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173. find
  18. Alec Fisher. 2004. The logic of real arguments. Cambridge University Press. find
  19. Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620. find
  20. R Edward Freeman. 2010. Strategic management: A stakeholder approach. Cambridge university press. find
  21. Carol Gilligan. 1993. In a different voice: Psychological theory and women’s development. Harvard university press. find
  22. Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The Jonathan Haidt. 2007. The new synthesis in moral psychology. science, 316(5827):998–1002. find
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275.
  24. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  25. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  26. Geert Hofstede. 2001. Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations. International Educational and Professional. find
  27. David Hume. 2000. A treatise of human nature. Oxford University Press. find
  28. Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, and Mor Geva. 2024. A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains. arXiv preprint arXiv:2402.00559.
  29. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Moralbench: Moral evaluation of llms. arXiv preprint arXiv:2406.04428. — [arXiv:2406.04428](https://arxiv.org/abs/2406.04428) find
  30. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  31. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, and 1 others. 2021. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574. find
  32. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny T Liang, Sydney Levine, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jack Hessel, and 1 others. 2025. Investigating machine moral judgement through the delphi experiment. Nature Machine Intelligence, pages 1–16. find
  33. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. find
  34. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  35. Olya Kudina, Brian Ballsun-Stanton, and Mark Alfano. 2025. The use of large language models as scaffolds for proleptic reasoning. Asian Journal of Philosophy, 4(1):1–18. find
  36. Shivani Kumar and David Jurgens. 2025. Are rules meant to be broken? understanding multilingual moral reasoning as a computational pipeline with unimoral. arXiv preprint arXiv:2502.14083. find
  37. Andrew K Lampinen, Nicholas Roy, Ishita Dasgupta, Stephanie CY Chan, Allison Tam, James Mcclelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane Wang, and 1 others. 2022. Tell me why! explanations support learning relational and causal structure. In International Conference on Machine Learning, p… find
  38. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  39. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. 2024. Self-alignment with instruction backtranslation. In ICLR.
  40. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  41. John Stuart Mill. 2016. Utilitarianism. In Seven masterpieces of philosophy, pages 329–375. Routledge. find
  42. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. Reframing instructional prompts to gptk’s language. arXiv preprint arXiv:2109.07830.
  43. Swaroop Mishra and Elnaz Nouri. 2023. Help me think: A simple prompting strategy for non-experts to create customized content with models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11834–11890. find
  44. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, and 1 others. 2021. Show your work: Scratchpads for intermediate computation with language models.
  45. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2024. 2 ol… find
  46. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744.
  47. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  48. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  49. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and 1 others. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.
  50. John Rawls. 2017. A theory of justice. In Applied ethics, pages 21–29. Routledge. find
  51. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675.
  52. Milton Rokeach. 1973. The nature of human values. Free press. find
  53. William David Ross. 2002. The right and the good. Oxford University Press. find
  54. Shalom H Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier. find
  55. Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, and 1 others. 2024. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial…
  56. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  57. George Tovstiga. 2023. What is first principles thinking? In Strategy Praxis: Insight-Driven, First Principles-Based Strategic Thinking, Analysis, and Decision-Making, pages 41–65. Springer. find
  58. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.
  59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  60. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  61. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  62. Meilan Zhang. 2013. Prompts-based scaffolding for online inquiry: Design intentions and classroom realities. Journal of Educational Technology & Society, 16(3):140–151. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakraborty-2025-structured-moral-reasoning-language-models/