Elamur Research⌕
☰
home / papers / 2025 / Can Language Models Reason about Individualistic Human Values and Preferences?

Can Language Models Reason about Individualistic Human Values and Preferences?

Paper · 2025 · Liwei Jiang, Taylor Sorensen, Sydney Levine et al.

Summary

The paper argues that pluralistic alignment usually sorts people into predefined groups such as demographics, personality, or culture. That approach can erase individual differences and invite stereotyping. The authors propose individualistic alignment instead and focus on one concrete part of it: individualistic value reasoning. Here a model sees some of a person's value statements and must infer how that person would judge new questions. To support this task, they convert the World Values Survey (WVS) into IndieValueCatalog. Its varied questions and answer formats become uniform first-person natural-language statements, available at two granularities (refined and polarity-grouped). Frontier LMs score only about 55–65% zero-shot. Adding demographic information to the value statements helps strong models very little, so group labels cannot replace individual evidence. The authors also introduce the Value Inequity Index (σINEQUITY), which shows that models perform unevenly across population groups. Finally, they fine-tune IndieValueReasoner models, which improve both accuracy and equity on held-out people. These experiments also reveal patterns in global human values.

Key points

  • Dataset scale: IndieValueCatalog covers 93,279 WVS respondents from 70+ countries.
    • 253 questions map to 929 refined or 567 polar statements.
    • Each person has on average about 242 value statements plus 31 demographic statements.
  • Evaluation setup:
    • Demonstrations range from 50 to 200 statements per person.
    • Probes cover 39 statements across 13 WVS categories.
    • Results are averaged over three cross-validation splits.
    • 800 demographically balanced individuals are held out for testing.
  • Baseline performance: Frontier LMs reach only 55–65% zero-shot accuracy at predicting an individual's value choices.
  • Role of demographics: Supplying demographic statements gives only marginal gains, which cautions against defining people by group membership.
  • Fairness and training: The Value Inequity Index (σINEQUITY) exposes uneven treatment across groups. IndieValueReasoner models, trained with Open-Instruct, improve both accuracy and σINEQUITY.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Liwei Jiang, Taylor Sorensen, Sydney Levine and Yejin Choi. 2025. Can Language Models Reason about Individualistic Human Values and Preferences?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.336

@inproceedings{jiang2025can,
  title = {{Can Language Models Reason about Individualistic Human Values and Preferences?}},
  author = {Liwei Jiang, Taylor Sorensen, Sydney Levine and Yejin Choi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.336},
  url = {https://doi.org/10.18653/v1/2025.acl-long.336}
}

References

61 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  2. Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. 2024. Persona: A reproducible testbed for pluralistic alignment. Preprint, arXiv:2407.17387. find
  3. Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. 2024a. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. Preprint, arXiv:2406.08469. — [arXiv:2406.08469](https://arxiv.org/abs/2406.08469) find
  4. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2024b. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. Preprint, arXiv:2309.13007.
  5. Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024a. Culturalteaming: Ai-assisted interactive red-teaming for challenging llms’ (lack of) multicultural knowledge. Preprint, arXiv:2404.06664. — [arXiv:2404.06664](https://arxiv.org/abs/2404.06664) find
  6. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024b. Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. Preprint… — [arXiv:2410.02677](https://arxiv.org/abs/2410.02677) find
  7. Oliver Scott Curry, Matthew Jones Chesters, and Caspar J Van Lissa. 2019a. Mapping morality with a compass: Testing the theory of ‘morality-as-cooperation’with a new questionnaire. Journal of Research in Personality, 78:106–124. find
  8. Oliver Scott Curry, Daniel Austin Mullins, and Harvey Whitehouse. 2019b. Is it good to cooperate? testing the theory of morality-as-cooperation in 60 societies. Current anthropology, 60(1):47–69. find
  9. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring t…
  10. Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration. Preprint, arXiv:2406.15951.
  11. Yi Ren Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. ArXiv, abs/2402.09369.
  12. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, José Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Björn Puranen, editors. 2020a. World Values Survey: Round Seven – Country-Pooled Datafile. JD Systems Institute and WVSA Secretariat, Madrid,… find
  13. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Björn Puranen, editors. 2020b. World Values Survey: Round Seven – Country-Pooled Datafile. JD Systems Institute & WVSA Secretariat, Madrid, Sp… find
  14. Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. 2024. Value augmented sampling for language model alignment and personalization. Preprint, arXiv:2405.06639. — [arXiv:2405.06639](https://arxiv.org/abs/2405.06639) find
  15. Nien-hê Hsieh and Henrik Andersson. 2021. Incommensurable Values. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Fall 2021 edition. Metaphysics Research Lab, Stanford University. find
  16. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. Preprint, arXiv:2310.11564. find
  17. Kaixuan Ji, Jiafan He, and Quanquan Gu. 2024. Reinforcement learning from human feedback with active queries. Preprint, arXiv:2402.09401. — [arXiv:2402.09401](https://arxiv.org/abs/2402.09401) find
  18. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023. Evaluating and inducing personality in pre-trained language models. In Thirty-seventh Conference on Neural Information Processing Systems. find
  19. Kyriaki Kalimeri, Mariano G. Beiró, Matteo Delfino, Robert Raleigh, and Ciro Cattuto. 2019. Predicting demographics, moral foundations, and human values from digital behaviours. Computers in Human Behavior, 92:428–445. find
  20. Vijay Keswani, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, and Walter Sinnott-Armstrong. 2024. On the pros and cons of active learning for moral preference elicitation. Preprint, arXiv:2407.18889. — [arXiv:2407.18889](https://arxiv.org/abs/2407.18889) find
  21. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024a. The prism alignment project: What participatory, representative and individualised human feedback reveals ab… find
  22. Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Kohen Behar, and David M. Chan. 2024. Virtual personas for language models via an anthology of backstories. Preprint, arXiv:2407.06576. find
  23. H.R. Kirk, B. Vidgen, P. Röttger, et al. 2024b. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, 6:383–392. find
  24. Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. 2023. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers. Preprint, arXiv:2311.09180. find
  25. Louis Kwok, Michal Bravansky, and Lewis Griffin. 2024. Evaluating cultural adaptability of a large language model via simulation of synthetic personas. In First Conference on Language Modeling. find
  26. Thom Lake, Eunsol Choi, and Greg Durrett. 2024. From distributional to overton pluralism: Investigating large language model alignment. Preprint, arXiv:2406.17692. find
  27. Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. 2024. Aligning to thousands of preferences via system message generalization. Preprint, arXiv:2405.17977. — [arXiv:2405.17977](https://arxiv.org/abs/2405.17977) find
  28. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. Culturellm: Incorporating cultural differences into large language models. ArXiv, abs/2402.10946.
  29. Junyi Li, Charith Peris, Ninareh Mehrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. 2024b. The steerability of large language models toward data-driven personas. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational… find
  30. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2024. Llms + persona-plug = personalized llms. Preprint, arXiv:2409.11901.
  31. William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. 2024. Active preference learning for large language models. Preprint, arXiv:2402.08114. — [arXiv:2402.08114](https://arxiv.org/abs/2402.08114) find
  32. Jun-Hee Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Pérez-Almendros, Abinew Ali Ayele, V’ictor Guti’errez-Basulto, Yazm’in Ib’anez-Garc’ia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Rzayev, Nina White, Seid Muhie… — BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
  33. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  34. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST ’22, New Yo… find
  35. Pew Research Center. n.d. Pew research center. https://www.pewresearch.org. Accessed: 2024-09-30. ↗
  36. Gregory R. Maio. 2010. Chapter 1 - mental representations of social values. In Advances in Experimental Social Psychology, volume 42 of Advances in Experimental Social Psychology, pages 1–43. Academic Press. find
  37. Wasu Top Piriyakulkij, Volodymyr Kuleshov, and Kevin Ellis. 2024. Active preference inference using language models and probabilistic reasoning. Preprint, arXiv:2312.12009. find
  38. Elinor Mason. 2023. Value Pluralism. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy, Summer 2023 edition. Metaphysics Research Lab, Stanford University. find
  39. Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2024. Personalizing reinforcement learning from human feedback with variational preference learning. Preprint, arXiv:2408.10075. — [arXiv:2408.10075](https://arxiv.org/abs/2408.10075) find
  40. Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. 2023. Sample efficient reinforcement learning from human feedback via active exploration. Preprint, arXiv:2312.00267. find
  41. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  42. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. ArXiv, abs/2404.12464. — [arXiv:2404.12464](https://arxiv.org/abs/2404.12464) find
  43. Michael Ryan, William Held, and Diyi Yang. 2024. Unintended impacts of LLM alignment on global representation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16121–16140, Bangkok, Thailand. Association for Computational Lingu…
  44. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  45. Jonathan Schaffer. 2018. Monism. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Winter 2018 edition. Metaphysics Research Lab, Stanford University. find
  46. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  47. Shalom H. Schwartz. 2012. An overview of the schwartz theory of basic values. Online Readings in Psychology and Culture, 2(1). find
  48. Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. Preprint, arXiv:2307.00184.
  49. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua yu, Raya Horesh, Rogério Abreu de Paula, and Diyi Yang. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. Preprint, arXiv:2404.15238.
  50. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. A roadmap to pluralistic alignment. Preprint, arXiv:2402.05070.
  51. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. Preprint, arXiv:2404.18796.
  52. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180–5197, Dublin, Ireland. Association for Computational Linguistics. find
  53. Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. 2024. Self-exploring language models: Active preference elicitation for online alignment. Preprint, arXiv:2405.19332. — [arXiv:2405.19332](https://arxiv.org/abs/2405.19332) find
  54. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. World-ValuesBench: A large-scale benchmark dataset for multi-cultural value awareness of language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Reso… find
  55. Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. Sotopia: Interactive evaluation for social intelligence in language agents. Preprint, arXiv:2310.11667.
  56. Minjun Zhu, Linyi Yang, and Yue Zhang. 2024. Personality alignment of large language models. Preprint, arXiv:2408.11779.
  57. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291.
  58. Michael J. Zimmerman and Ben Bradley. 2019. Intrinsic vs. Extrinsic Value. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Spring 2019 edition. Metaphysics Research Lab, Stanford University. find
  59. Paul Stenner, S. Watts, and M. Worrell. 2008. Q Methodology, pages 215–239. Sage Research Methods. find
  60. Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R. Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji. 2024. Persona-db: Efficient large language model personalization for response prediction with collaborative data refinement. Preprint, arXiv:2402.11060.
  61. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2025-can-language-models-reason-about/