Elamur Research⌕
☰
home / papers / 2024 / Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

Paper · 2024 · Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang et al.

Summary

The paper examines whether large language models, trained mostly on English text, push English-speaking (largely American) culture onto users who ask questions in other languages. For example, they may name Thanksgiving when asked about public holidays in Chinese. To measure this "cultural dominance," the authors built a benchmark with two parts. The concrete part asks models to list ten items in eight categories (holidays, songs, books, movies, celebrities, heroes, history, mountains) in eleven languages. Each item is labelled by cultural origin using Wikipedia, giving an In-Culture Score. The abstract part uses the official multilingual versions of the World Values Survey and the Political Coordinates Test in six languages. It checks whether model answers resemble human answers from the matching culture or English-language results. GPT-family models showed clear English bias. GPT-4 was the most affected and text-davinci-003 the least. Two fixes helped: pretraining on more diverse, non-English-heavy data, and prompts that explicitly name the target culture. The prompting fix worked mainly for concrete items. The study matters because it shows that ordinary non-English users may receive culturally mismatched answers.

Key points

  • The concrete benchmark uses the template "Please list 10 {OBJECT} for me." Besides English, it spans ten further languages, which in alphabetical order are Arabic, Chinese, French, German, Hindi, Indonesian, Italian, Japanese, Korean and Russian.
  • Scoring works as follows:
    • Each of the 10 returned items earns a point if Wikipedia ties it to the culture of the query language.
    • Items shared across cultures, such as New Year's Day, count for every relevant language.
  • The abstract evaluation uses two surveys and six languages (English, Chinese, Russian, Indonesian, Hindi, Arabic):
    • WVS, latest wave 2017–2020 across 57 countries, with traditional/secular and survival/self-expression axes.
    • PCT, a 36-item quiz with economic and social axes.
  • Model outputs in language l are compared with three targets: H_ref (humans in that language), H_en (English-speaking humans) and M_en (the model's own English output).
  • Culture-aware prompting, such as asking for answers "in the culture of [lang]", fixes much of the bias for concrete objects but helps less for values and opinions. Diverse pretraining helps more broadly but costs more.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu and Michael Lyu. 2024. Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.345

@inproceedings{wang2024not,
  title = {{Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models}},
  author = {Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu and Michael Lyu},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.345},
  url = {https://doi.org/10.18653/v1/2024.acl-long.345}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This paper has two primary limitations that offer avenues for future research. find
  2. Walter L. Adamson. 1980. Hegemony and revolution: A study of antonio gramsci’s political and cultural theory. find
  3. The first limitation pertains to the range of concrete cultural objects examined: we have only considered eight such objects, spanning eleven languages. This relatively narrow scope invites the extension of subsequent research to a broader spectrum of objects and languages, enhancing the comprehensi… find
  4. Gati Aher, RosaI. Arriaga, and Adam Tauman Kalai. 2022. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning. find
  5. The second limitation relates to our reliance on existing public surveys from the social sciences to study abstract values and opinions. The potential bias inherent in these surveys’ scope and Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiez… our card
  6. Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeology of books known to chatgpt/gpt-4. ArXiv, abs/2305.00118. our card
  7. Christof Demont-Heinrich. 2011. Cultural imperialism versus globalization of culture: Riding the structure-agency dialectic in global communication and media studies. Sociology Compass, 5:666–678. find
  8. J. Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. find
  9. Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan,… our card
  10. Shuzheng Gao, Xinjie Wen, Cuiyun Gao, Wenxuan Wang, and Michael R. Lyu. 2023. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. ArXiv, abs/2304.07575. and Yaodong Yang. 2023a. Beavertails: Towards improved safety alignment of llm via a human-preference… our card
  11. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023b. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852. find
  12. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. ArXiv, abs/2301.08745. our card
  13. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785. find
  14. Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870. — [arXiv:2307.07870](https://arxiv.org/abs/2307.07870) find
  15. T. J. Jackson Lears. 1985. The concept of cultural hegemony: Problems and possibilities. The American Historical Review, 90:567–593. find
  16. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. ArXiv, abs/2301.01768. find
  17. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342. our card
  18. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Annual Meeting of the Association for Computational Linguistics. our card
  19. Youssef Mohamed, Mohamed AbdelFattah, Shyma Alhuwaider, Li Fei-Fei, Xiangliang Zhang, Kenneth Ward Church, and Mohamed Elhoseiny. 2022. Artelingo: A million emotion annotations of wikiart with emphasis on diversity over language and culture. EMNLP. find
  20. Zhiwei He, Ti Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring human-like translation strategy with large language models. ArXiv, abs/2305.04118. our card
  21. Cas Mudde. 2013. Three decades of populist radical right parties in western europe: So what? European Journal of Political Research, 52(1):1–19. find
  22. Jen-tse Huang, Man Ho Adrian Lam, Eric Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R. Lyu. 2023. Emotionally numb or empathetic? evaluating how llms feel using emotionbench. ArXiv, abs/2308.03656. — [arXiv:2308.03656](https://arxiv.org/abs/2308.03656) find
  23. Ronald Inglehart, Miguel Basanez, Jaime Diez-Medrano, Loek Halman, and Ruud Luijkx. 2000. World values surveys and european values surveys, 1981-1984, 1990-1993, and 1995-1997. Ann Arbor-Michigan, Institute for Social Research, ICPSR version. find
  24. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, Tarek Naous, Michael J. Ryan, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. ArXiv, abs/2305.14456. our card
  25. OpenAI. 2023. Gpt-4 system card. find
  26. Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069. — [arXiv:2211.13069](https://arxiv.org/abs/2211.13069) find
  27. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? ArXiv, abs/2303.17548. our card
  28. Susan C. Seymour. 2006. Resistance. Anthropological Theory, 6:303 – 321. find
  29. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of chinese large language models. ArXiv, abs/2304.10436. find
  30. Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R. Lyu. 2023. Biasasker: Measuring the bias in conversational ai system. ArXiv, abs/2305.12434. — [arXiv:2305.12434](https://arxiv.org/abs/2305.12434) find
  31. Shuohuan Wang, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shi Feng, Junyuan Shang, Yanbin Zhao, Chao Pang, Jiaxiang Liu, Xuyi Chen, Yuxiang Lu, Weixin Liu, Xi Wang, Yangfan Bai, Qiuliang Chen, Li Zhao, Shiyong Li, Peng Sun, Dianhai Yu, Yanjun Ma, Hao Tian, Hua Wu, Tian Wu, Wei Zeng, Ge L… find
  32. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2023. All languages matter: On the multilingual safety of large language models. arXiv preprint arXiv:2310.00905. — [arXiv:2310.00905](https://arxiv.org/abs/2310.00905) find
  33. Jeanette Haynes Writer. 2008. Unmasking, exposing, and confronting: Critical race theory, tribal critical race theory and multicultural education. International Journal of Multicultural Education, 10. find
  34. Hao Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael R. Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. ArXiv, abs/2303.13648. find
  35. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463. our card
  36. A Discussion on Language and Culture A Discussion on Language and Culture find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-not-all-countries-celebrate-thanksgiving/