Elamur Research
☰
home / papers / 2024 / Investigating Cultural Alignment of Large Language Models

Investigating Cultural Alignment of Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.acl-long.671.pdf

Summary

The paper asks whether large language models reflect the views held by specific cultures and what drives that alignment. The authors simulate sociological surveys. Each model receives the demographic profile, or "persona," of a real World Values Survey (WVS-7) respondent from Egypt or the United States and must choose an answer. That choice is then compared with the person's actual reply.

Thirty intentionally non-trivial questions, each rephrased four ways, are asked in English and Arabic. Four models with different pretraining language mixes are tested: GPT-3.5, mT0-XXL, LLaMA-2-13B-Chat and its Arabic-oriented derivative AceGPT-13B-Chat.

Alignment improves when the prompt uses a culture's dominant language and when pretraining includes more of the languages that culture uses. Misalignment grows for digitally underrepresented personas and for sensitive themes such as social values. The authors also introduce Anthropological Prompting, which has the model reason about the persona before answering, to raise alignment. The study calls for more balanced multilingual pretraining data and connects cultural alignment to questions of cross-lingual transfer.

Key points

  • Survey data: answers come from WVS-7 (fielded 2017–2021, 259 questions in total), using 1,200 Egyptian and 2,596 US participants.
  • Personas: each persona is defined by six attributes: region, sex, age, social class, education level and marital status.
  • Questions: 30 questions were chosen, with four paraphrases each generated by ChatGPT. The Arabic versions were machine-translated and then corrected by native speakers.
  • Models: gpt-3.5-turbo-1106, mT0-XXL, LLaMA-2-13B-Chat and AceGPT-13B-Chat (LLaMA-2 further finetuned on Arabic and English). The LLaMA-2/AceGPT pair lets the authors examine cross-lingual transfer through finetuning.
  • Terminology and release: the authors prefer "Cultural Trends" over "bias." Code and data are released at github.com/bkhmsi/cultural-trends.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.671

@inproceedings{alkhamissi2024investigating,
  title = {{Investigating Cultural Alignment of Large Language Models}},
  author = {AlKhamissi, Badr and ElNokrashy, Muhammad and Alkhamissi, Mai and Diab, Mona},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.671},
  url = {https://doi.org/10.18653/v1/2024.acl-long.671}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 298–306, New York, NY, USA. Association for Computing Machinery. find our card
  2. Zaid Alyafeai, Maged S. Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating arabic nlp tasks using chatgpt models. ArXiv, abs/2306.16322. find
  3. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguis… find
  4. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, Dubrovnik, Croatia. As… find our card
  5. James Clifford, Kim Fortun, Marcus George E., Clifford James, George E. Marcus, Pratt Mary Louise, Fischer Michael M. J., Rabinow Paul, Rosaldo Renato, Tyler Stephen A., Asad Talal, and Crapanzano Vincent. 2020. Writing culture. find
  6. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236–1270, Singapore. Association for Computational Ling… find our card
  7. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. Towards measuring t… find our card
  8. Negar Foroutan, Mohammadreza Banaei, Rémi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discovering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7560–7575, Abu Dhabi, United Arab Emi… find
  9. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. 2020. World values survey wave 7 (2017-2020) cross-national data-set. find
  10. G. Hofstede. 1984. Culture’s Consequences: International Differences in Work-Related Values. Cross Cultural Research and Methodology. SAGE Publications. find
  11. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic… find
  12. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online. Associ… find our card
  13. Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. 2023. Personas as a way to model truthfulness in language models. ArXiv, abs/2310.18168. find
  14. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. find
  15. A. L. Kroeber and Clyde Kluckhohn. 1952. Culture: A Critical Review of Concepts and Definitions. Peabody Museum Press, Cambridge, Massachusetts. find
  16. Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen. 2023. Improving diversity of demographic representation in large language models via collective-critiques and self-voting. In Proce… find
  17. Bryan Li, Samar Haider, and Chris Callison-Burch. 2024. This land is {Your, My} land: Evaluating geopolitical biases in language models through territorial disputes. 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  18. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… find our card
  19. Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics. find
  20. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… find our card
  21. Tarek Naous, Michael Joseph Ryan, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. ArXiv, abs/2305.14456. find our card
  22. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Nationality bias in text generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 116–122, Dubrovnik, Croatia. Association f… find
  23. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
  24. Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666, Singapore. Association for Computational Linguistics. find
  25. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find our card
  26. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find our card
  27. Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2023. Do llms exhibit human-like response biases? a case study in survey design. ArXiv, abs/2311.04076. find
  28. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… find
  29. Edward B. Tylor. 1871. Primitive culture : researches into the development of mythology, philosophy, religion, art, and custom, 3rd ed., rev edition. John Murray London, London. find
  30. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023. Instruction tuning for large language models: A survey. ArXiv, abs/2308.10792. find our card
  31. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. The language barrier: Dissecting safety challenges of llms in multilingual contexts. ArXiv, abs/2401.13136. find
  32. Model English GPT-3.5 AceGPT-Chat LLaMA-2-Chat mT0-XXL Egypt 52.69 / 30.17 49.19 / 31.74 52.92 / 31.67 48.52 / 31.86 United States English Arabic Arabic 53.45 / 32.92 52.35 / 33.55 48.97 / 28.18 47.81 / 29.16 65.26 / 41.52 54.79 / 32.37 63.69 / 39.52 53.73 / 31.42 62.74 / 39.72 51.20 / 27.47 61.02 /… find