Elamur Research⌕
☰
home / papers / 2025 / Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?

Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?

Paper · 2025 · Hua Shen, Nicholas Clark and Tanu Mitra

Summary

Most evaluations of LLM values only ask models which values they endorse. They rarely check whether the models' behaviour in concrete situations matches those claims. Borrowing the "value-action gap" idea from social psychology, the authors built ValueActionLens to measure this mismatch. Using Schwartz's theory of human values, they created the VIA dataset. It covers 132 scenarios, pairing 12 countries with 11 social topics such as health, religion and politics. Each scenario includes LLM-generated actions, with explanations, that express agreement or disagreement with 56 values. Human reviewers checked the generated actions.

Models complete two tasks. First, they rate how much they endorse a value in a given context. Second, they pick which of two actions fits that value. Three metrics then compare the two answers. Across six LLMs, the authors found large inconsistencies between stated values and chosen actions, and the size of the gap depended on value, culture, topic and model. The results suggest that self-reported values are an unreliable proxy for model behaviour. They also argue for context-sensitive, pluralistic alignment checks.

Key points

  • VIA contains 14,784 value-informed actions: 12 countries × 11 topics × 56 values × 2 inclinations (Agree/Disagree). Each action comes with feature attributions and a natural-language explanation.
  • The three alignment measures are value-action alignment rate, alignment distance (Manhattan/Euclidean) and alignment ranking.
  • GPT-4o-mini, Claude-Sonnet-4 and Llama-3.3-70B generally aligned worse in African and Asian contexts than in North American and European ones.
  • In qualitative cases, a model claimed to reject "Social Power" yet chose dominating actions (India/work, Germany/politics). It also expressed loyalty but acted against it in a US religious context. GPT-4o-mini handled "Choosing Own Goals" poorly for the Philippines but well for the U.S.
  • Data and code are available at github.com/huashen218/value_action_gap. Stated limitations include binary inclinations, forced-choice actions and no evaluation of dialogue settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksSocial Sciences · 1,772 worksSafety Research · 543 worksDecision Sciences · 209 worksPhilosophy · 284 worksInformation Systems and Management · 41 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hua Shen, Nicholas Clark and Tanu Mitra. 2025. Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.154

@inproceedings{shen2025mind,
  title = {{Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?}},
  author = {Hua Shen, Nicholas Clark and Tanu Mitra},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.154},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.154}
}

References

50 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. 2024. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614. — [arXiv:2402.04614](https://arxiv.org/abs/2402.04614) find
  3. Icek Ajzen. 1980. Understanding attitudes and predicti-ing social behavior. Englewood cliffs. find
  4. Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, and Pascale Fung. 2024. High-dimension human value representation in large language models. arXiv preprint arXiv:2404.07900. find
  5. Shan-Shan Chung and Monica Miu-Yin Leung. 2007. The value-action gap in waste recycling: The case of undergraduates in hong kong. Environmental Management, 40:603–612. find
  6. Preetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury, and Tanushree Mitra. 2024. " they are uncultured": Unveiling covert harms and social threats in llm generated conversations. arXiv preprint arXiv:2405.05378. find
  7. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  8. Public-Use Microdata File. 2017. General social survey. find
  9. Martin Fishbein and Icek Ajzen. 1980. Predicting and understanding consumer behavior: Attitude-behavior correspondence. Understanding attitudes and predicting social behavior, 1(1):148–172. find
  10. Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437. find
  11. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, and 1 others. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv:2209.07858.
  12. Gaston Godin, Mark Conner, and Paschal Sheeran. 2005. Bridging the intention–behaviour gap: The role of moral norm. British journal of social psychology, 44(4):497–512. find
  13. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  14. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, K Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, Bi Puranen, and 1 others. 2020. World values survey: Round seven-country-pooled datafile. madrid, spain & vienna, austria: Jd systems institute & wvsa… find
  15. Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2023. How (not) to use sociodemographic information for subjective nlp tasks. arXiv preprint arXiv:2309.07034. — [arXiv:2309.07034](https://arxiv.org/abs/2309.07034) find
  16. Bahareh Harandizadeh, Abel Salinas, and Fred Morstatter. 2024. Risk and response in large language models: Evaluating key threat categories. arXiv preprint arXiv:2403.14988. — [arXiv:2403.14988](https://arxiv.org/abs/2403.14988) find
  17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. resource management: Understanding why employees resist algorithmic evaluation at workpla…
  18. Han Jiang, Xiaoyuan Yi, Zhihua Wei, Ziang Xiao, Shu Wang, and Xing Xie. 2024a. Raising the bar: Investigating the values of large language models via generative evolving testing. arXiv preprint arXiv:2406.14230. — [arXiv:2406.14230](https://arxiv.org/abs/2406.14230) find
  19. Prolific. 2024. Prolific. https://www.prolific.com. First released in 2014. Current version used: [insert current month(s) and year(s) of use]. ↗
  20. Liwei Jiang, Sydney Levine, and Yejin Choi. 2024b. Can language models reason about individualistic human values and preferences? In Pluralistic Alignment Workshop at NeurIPS 2024.
  21. Florian G Kaiser, Sybille Wölfing, and Urs Fuhrer. 1999. Environmental attitude and ecological behaviour. Journal of environmental psychology, 19(1):1–19. find
  22. Dinesh Kalla, Nathan Smith, Fnu Samaah, and Sivaraju Kuraku. 2023. Study and analysis of chat gpt and its impact on different fields of study. International journal of innovative science and research technology, 8(3). find
  23. Elise Karinshak, Amanda Hu, Kewen Kong, Vish-wanatha Rao, Jingren Wang, Jindong Wang, and Yi Zeng. 2024. Llm-globe: A benchmark evaluating the cultural values embedded in llm output. arXiv preprint arXiv:2411.06032. — [arXiv:2411.06032](https://arxiv.org/abs/2411.06032) find
  24. Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. 2024. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. arXiv preprint arXiv:2406.14805. — [arXiv:2406.14805](https://arxiv.org/abs/2406.14805) find
  25. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. 2024. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1–10. find
  26. Siyang Liu, Trisha Maturi, Bowen Yi, Siqi Shen, and Rada Mihalcea. 2024. The generation gap: Exploring age bias in the value systems of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19617–19634. find
  27. Chung Kwan Lo. 2023. What is the impact of chatgpt on education? a rapid review of the literature. Education Sciences, 13(4):410. find
  28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  29. Hyanghee Park, Daehwan Ahn, Kartik Hosanagar, and Joonhwan Lee. 2021. Human-ai interaction in human Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. arXiv preprint arXiv… — [arXiv:2406.04214](https://arxiv.org/abs/2406.04214)
  30. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. arXiv preprint arXiv:2402.16786.
  31. Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R Brubaker. 2021. A framework of severity for harmful content online. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–33. find
  32. Shalom H Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier. find
  33. Shalom H Schwartz. 1994. Are there universal aspects in the structure and contents of human values? Journal of social issues, 50(4):19–45. find
  34. Shalom H Schwartz. 2005. Robustness and fruitfulness of a theory of universals in individual values. Valores e trabalho, pages 56–85. find
  35. Shalom H Schwartz. 2012. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):11. find
  36. Pola Schwöbel, Luca Franceschi, Muhammad Bilal Zafar, Keerthan Vasist, Aman Malhotra, Tomer Shenhar, Pinal Tailor, Pinar Yilmaz, Michael Diamond, and Michele Donini. 2024. Evaluating large language models with fmeval. arXiv preprint arXiv:2407.12872. — [arXiv:2407.12872](https://arxiv.org/abs/2407.12872) find
  37. Pola Schwöbel, Jacek Golebiowski, Michele Donini, Cédric Archambeau, and Danish Pruthi. 2023. Geographical erasure in language generation. arXiv preprint arXiv:2310.14777. find
  38. Hua Shen, Chieh-Yang Huang, Tongshuang Wu, and Ting-Hao Kenneth Huang. 2023. Convxai: Delivering heterogeneous ai explanations via conversations to support human-ai scientific writing. In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, pages… find
  39. Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, and 1 others. 2024a. Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions. arXiv preprint arXiv:2406.09… find
  40. Hua Shen, Tiffany Knearem, Reshmi Ghosh, Yu-Ju Yang, Tanushree Mitra, and Yun Huang. 2024b. Valuecompass: A framework of fundamental values for human-ai alignment. arXiv preprint arXiv:2409.09586. — [arXiv:2409.09586](https://arxiv.org/abs/2409.09586) find
  41. Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. 2020. Interpretable deep learning under fire. In 29th {USENIX} security symposium ({USENIX} security 20). find
  42. Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, and Yaodong Yang. 2025. Heterogeneous value alignment evaluation for large language models. In International Conference on Artificial General Intelligence, pages 381–392. Springer. find
  43. Hua Shen, Yuguang Yang, Guoli Sun, Ryan Langman, Eunjung Han, Jasha Droppo, and Andreas Stolcke. 2022. Improving fairness in speaker verification via group-adapted fusion network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7077– 7081.… find
  44. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, and 1 others. 2024. A roadmap to pluralistic alignment. arXiv:2402.05070.
  45. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  46. Iris Vermeir and Wim Verbeke. 2006. Impact of values, involvement and perceptions on consumer attitudes and intentions towards sustainable consumption. Journal of Agricultural and Environmental Ethics, 19(2). find
  47. Qiaosi Wang, Michael Madaio, Shaun Kane, Shivani Kapania, Michael Terry, and Lauren Wilcox. 2023. Designing responsible ai: Adaptations of ux practice to meet responsible ai challenges. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–16. find
  48. Kyra Wilson and Aylin Caliskan. 2024. Gender, race, and intersectional bias in resume screening via language model retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1578–1590. find
  49. Haoran Ye, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. 2025a. Large language model psychometrics: A systematic review of evaluation, validation, and enhancement. arXiv preprint arXiv:2505.08245. — [arXiv:2505.08245](https://arxiv.org/abs/2505.08245) find
  50. Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. 2025b. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26400–26408. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2025-mind-value-action-gap-do/