Elamur Research
☰
home / papers / 2023 / Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study

Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.c3nlp-1.7.pdf

Summary

ChatGPT serves users in many countries and learned from multilingual data that carries many cultural norms, yet it is unclear whose values its answers reflect. The authors build a probing framework around the Hofstede Culture Survey, a widely used cross-cultural instrument. They rewrite its 24 items from second to third person so the model is not assigned a personal identity. They then pose the items to ChatGPT for five cultures (US, China, Germany, Japan, Spain) using three templates: two English prompts that name the nationality, and one in the country's main language. The answers are converted into six cultural dimension scores and compared with human survey data using Spearman correlation. The authors also check consistency across prompts and test a multi-turn knowledge-injection strategy. ChatGPT aligns best with American values and adapts poorly to the other cultures. English prompts make its answers more uniform and push them toward an American profile, while native-language prompts usually improve alignment. The study shows that dialogue agents need more cultural diversity and awareness.

Key points

  • The six Hofstede dimensions used are pdi, idv, uai, mas, lto and ivr. Each score is a weighted combination of differences between 4 of the 24 survey answers, plus a constant.
  • Consistency between the two English prompts: answers matched 79.17% of the time for US, 70.83% for DE, JP and ES, and only 58.33% for CN.
  • Consistency between English and native-language prompts: CN 79.17%, DE 75.00%, ES 58.33%, JP 41.67%.
  • Case study: for Japan, time for personal or home life was rated "of utmost importance" (score 1) under the English prompt but only moderately important (score 3) under the Japanese prompt. Chinese questions showed similar gaps.
  • Knowledge injection: the multi-turn strategy tests three kinds of injected information (valid, meaningless and anti-factual). ChatGPT tended to accept false information over its own knowledge.
  • Limitations: the analysis treats language as a stand-in for culture. The authors also cannot rule out that the survey appeared in the training data, and they note that about 96% of InstructGPT's training corpus is English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 112 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.c3nlp-1.7

@inproceedings{cao2023assessing,
  title = {{Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study}},
  author = {Cao, Yong and Zhou, Li and Lee, Seolhwa and Cabello, Laura and Chen, Min and Hershcovich, Daniel},
  booktitle = {Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)},
  year = {2023},
  doi = {10.18653/v1/2023.c3nlp-1.7},
  url = {https://doi.org/10.18653/v1/2023.c3nlp-1.7}
}

References

42 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Seattle, U.S.A. Association for Computational Linguistics. find
  2. Muneer M. Alshater. 2022. Exploring the role of artificial intelligence in enhancing academic performance: A case study of ChatGPT. Available at SSRN. find
  3. Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does ChatGPT do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment. medRxiv, pages 2022–12. find
  4. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2022. Probing pre-trained language models for cross-cultural differences in values. arXiv preprint arXiv:2203.13722. find
  5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. find our card
  6. Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology, 101(2):366. find
  7. C. Haerpfer, R. Inglehart, A. Moreno, C. Welzel, K. Kizilova, Diez-Medrano J., P. Norris M. Lagos and, E. Ponarin, and B. Puranen. 2022. World values survey: Round seven - country-pooled datafile version 5.0. JD Systems Institute & WVSA Secretariat. find
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  9. Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin A Rothkopf, Alexander Fraser, and Kristian Kersting. 2022. Speaking multiple languages affects the moral bias of language models. arXiv preprint arXiv:2211.07733. find
  10. Census Bureau. 2020. Age by language spoken at home by ability to speak english for the population 5 years and over: 2011–2015 american community survey 5-year estimates. find
  11. Laura Cabello, Jiaang Li, and Ilias Chalkidis. 2023. Pokemonchat: Auditing chatgpt for pokémon universe knowledge. Available at SSRN. find
  12. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural N… find our card
  13. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  14. Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational L… find
  15. Geert Hofstede. 1984. Culture’s consequences: International differences in work-related values, volume 5. sage. find
  16. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. find
  17. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online. Associ… find our card
  18. Constanza Fierro and Anders Søgaard. 2022. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics. find
  19. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. arXiv preprint arXiv:2110.07574. find
  20. Kathleen C. Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi’s moral philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 26–42, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, an… find our card
  21. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785. find
  22. Candace Ross, Boris Katz, and Andrei Barbu. 2021. Measuring social biases in grounded vision and language embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 998–1008, Online. Association… find
  23. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? on opportunities and challenges of large language models for education. find our card
  24. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258–268. find
  25. Clyde Kluckhohn and Orval Hobart Mowrer. 1944. "culture and personality": A conceptual scheme. American Anthropologist, 46(1):1–29. find
  26. Richard A Shweder, Jacqueline J Goodnow, Giyoo Hatano, Robert A LeVine, Hazel R Markus, and Peggy J Miller. 2007. The cultural psychology of development: One mind, many mentalities. Handbook of Child Psychology, 1. find
  27. Yizhi Li, Ge Zhang, Bohao Yang, Chenghua Lin, Anton Ragni, Shi Wang, and Jie Fu. 2022. HERB: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 334– 346, Online only. Association for Computational… find
  28. Anders Søgaard. 2022. Should we ban English NLP for a year? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5254–5260, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  29. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana,… find
  30. Charles Spearman. 1961. The proof and measurement of association between two things. find
  31. Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael Wick. 2022. Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35… find
  32. Brady D Lund and Ting Wang. 2023. Chatting about ChatGPT: how may AI and GPT impact academia and libraries? Library Hi Tech News. find
  33. Gail M Sullivan and Richard Feinn. 2012. Using effect size—or why the p value is not enough. Journal of graduate medical education, 4(3):279–282. find
  34. Weicheng Ma, Samiha Datta, Lili Wang, and Soroush Vosoughi. 2022. EnCBP: A new benchmark dataset for finer-grained cultural background prediction in English. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2811–2823, Dublin, Ireland. Association for Computational Lingui… find
  35. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  36. Robert W McGee. 2023. Is Chat GPT biased against conservatives? an empirical study. Available at SSRN. find
  37. Cristina Mora. 2013. Cultures and organizations: Software of the mind intercultural cooperation and its importance for survival. Journal of Media Research, 6(1):65. find
  38. H Holden Thorp. 2023. ChatGPT is fun, but not an author. find
  39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems. find
  40. Marc Tomlinson, David Bracewell, and Wayne Krug. 2014. Capturing cultural differences in expressions of intentions. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 48–57, Dublin, Ireland. Dublin City University and Association fo… find
  41. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. find our card
  42. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15, Punta Cana, Dominican Republic. Association for Computat… find