Elamur Research⌕
☰
home / papers / 2024 / CulturePark: Boosting Cross-cultural Understanding in Large Language Models

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Paper · 2024 · Cheng Li, Damien Teney, Linyi Yang et al.

Summary

Large language models tend to favor Western, English-dominated perspectives because their training data underrepresents other cultures. Existing cultural datasets are mined from surveys, Wikipedia or social media, which makes them expensive to annotate and hard to scale. The authors propose CulturePark, a multi-agent framework built on GPT-3.5-turbo. It draws on Cognitive Conflict Theory and Social Cognitive Theory. An English-speaking "main contact" agent talks with role-played delegates from other cultures. Their disagreements push the conversation toward deeper explanations, new questions and richer answers. The resulting dialogues are fact-checked and diversified, giving about 41K samples. These samples are used to fine-tune eight culture-specific models.

Evaluation covers three areas: content moderation, alignment with Hofstede's cultural dimensions, and a human study on learning about cultures. The GPT-3.5-based models match or beat GPT-4 across 41 moderation datasets and align more closely on VSM 13. Participants also learned better and were more satisfied than with GPT-4. The approach offers a cheaper, synthetic alternative to large-scale cultural data collection.

Key points

  • Cultures and data: the work spans eight cultures (Korean, Chinese, Bengali, Turkish, Arabic, German, Spanish and Portuguese), with roughly 41,000 generated samples. Code is available at github.com/Scarelette/CulturePark.
  • Content moderation: the fine-tuned models equal or exceed GPT-4 on 41 datasets and clearly beat it in 5 cultures on 26 tasks. Fine-tuned Llama-2-70b variants outperform the base model in all 8 cultures and all 7 task types.
  • Data quality: CulturePark data shows higher diversity gain than CultureLLM. It also outperforms data generated directly by GPT-4 or GPT-3.5 without agent dialogue. Mixing cultures and genders among agents produced the best training data.
  • Forgetting check: on BIG-Bench-Hard (21 tasks), general performance was largely preserved and sometimes improved after cultural fine-tuning.
  • Limitations: the work focuses mainly on high-resource cultures because benchmarks for low-resource ones are scarce. Results also depend on OpenAI API model versions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie and Jindong Wang. 2024. CulturePark: Boosting Cross-cultural Understanding in Large Language Models. arXiv:2405.15145.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.15145 · arXiv 2405.15145

@misc{li2024culturepark,
  title = {{CulturePark: Boosting Cross-cultural Understanding in Large Language Models}},
  author = {Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie and Jindong Wang},
  year = {2024},
  doi = {10.48550/arxiv.2405.15145},
  eprint = {2405.15145},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.15145}
}

References

84 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Amin Abbasi, Arash Ghafouri, Mahdi Firouzmandi, Hassan Naderi, and Behrouz Minaei Bidgoli. Persianllama: Towards building first persian large language model. arXiv preprint arXiv:2312.15713, 2023. aimansnigdha. Bangla-abusive-comment-dataset. https://github.com/aimansnigdha/Bangla-Abusive-C… — [arXiv:2312.15713](https://arxiv.org/abs/2312.15713) ↗
  2. Miguel Á Álvarez-Carmona, Estefanıa Guzmán-Falcón, Manuel Montes-y Gómez, Hugo Jair Escalante, Luis Villasenor-Pineda, Verónica Reyes-Meza, and Antonio Rico-Sulayes. Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets. In Notebook papers of 3rd sepl… find
  3. John R Anderson, Lynne M Reder, and Herbert A Simon. Situated learning and education. Educational researcher, 25(4):5–11, 1996. find
  4. Arjun Appadurai. Modernity at large: Cultural dimensions of globalization. U of Minnesota P, 1996. find
  5. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop…
  6. The detailed hyperparameter setting is shown in Table 6. We only conduct this pilot study to show that Llama2 also works in CulturePark. The results might be further improved with more GPU resources. Shiladitya Bhattacharya, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer,… find
  7. Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv:2202.00132, 2022. arXiv preprint Y Cao, L Zhou, S Lee, L Cabello, M Chen, and D Hershcovich. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arxiv. Preprint posted online on…
  8. Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6193–6202, 2020.
  9. Çağrı Çöltekin. A corpus of turkish offensive language on social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174–6184, Marseille, France, 2020. URL https://www.aclweb.org/anthology/2020.lrec-1.758. ↗
  10. Alex J Chan, José Luis Redondo García, Fabrizio Silvestri, Colm O’Donnel, and Konstantina Palla. Harmonizing global voices: Culturally-aware models for enhanced content moderation. arXiv preprint arXiv:2312.02401, 2023. — [arXiv:2312.02401](https://arxiv.org/abs/2312.02401) find
  11. Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J Jansen, and Joni Salminen. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203–6212, 2020.
  12. Richard A Cosier and Gerald L Rose. Cognitive conflict and goal conflict effects on task performance. Organizational behavior and human performance, 19(2):378–391, 1977. daanVeer. Korean hatespeech dataset. https://github.com/daanVeer/HateSpeech_dataset, 2020. ↗
  13. Angel Felipe Magnossao de Paula and Ipek Baris Schlicht. Ai-upv at iberlef-2021 detoxis task: Toxicity detection in immigration-related web news comments using transformers and statistical models. arXiv preprint arXiv:2111.04530, 2021. — [arXiv:2111.04530](https://arxiv.org/abs/2111.04530) find
  14. Rogers P. de Pelle and Viviane P. Moreira. Offensive comments in the brazilian web: a dataset and baseline results. In 6th Brazilian Workshop on Social Network Analysis and Mining (BraSNAM), 2017. find
  15. Werner Delanoy. What is culture. The Cambridge handbook of intercultural communication, pages 17–34, 2020. find
  16. Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046, 2023. find
  17. Elisabetta Fersini, Paolo Rosso, Maria Anzovino, et al. Overview of the task on automatic misogyny identification at ibereval 2018. Ibereval@ sepln, 2150:214–228, 2018. find
  18. Susan T Fiske and Shelley E Taylor. Social cognition. Mcgraw-Hill Book Company, 1991. find
  19. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369, 2024.
  20. Yi R Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. Normsage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. arXiv preprint arXiv:2210.08604, 2022. find
  21. Michael Minkov Geert Hofstede, Gert Jan Hofstede. Cultures and Organizations: Software of the Mind, Third Edition. McGraw Hill Professional, https://books.google.co.uk/books?id=7bYWmwEACAAJ, 2010. ↗
  22. Google. Gemini. https://deepmind.google/technologies/gemini/\#introduction, 2023. ↗
  23. HASOC. Hasoc2020. https://hasocfire.github.io/hasoc/2020/index.html, 2020. ↗
  24. Hofstede. Vsm13. https://geerthofstede.com/research-and-vsm/vsm-2013/, 2013. ↗
  25. F Husain. Osact4 shared task on offensive language detection: Intensive preprocessing-based approach. arxiv 2020. arXiv preprint arXiv:2005.07297, 2020.
  26. Zhuoren Jiang, Zhe Gao, Guoxiu He, Yangyang Kang, Changlong Sun, Qiong Zhang, Luo Si, and Xiaozhong Liu. Detect camouflaged spam content via stoneskipping: Graph and text joint embedding for chinese character variation representation. In Proceedings of the 2019 Conference on Empirical Methods in Nat… find
  27. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785, 2022a.
  28. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785, 2022b.
  29. Sanaa Kaddoura and Safaa Henno. Dataset of arabic spam and ham tweets. Data in Brief, 52(10990): 4, 2024. find
  30. Kaggle. 5k turkish tweets with incivil content. https://www.kaggle.com/datasets/kbulutozler/5k-turkish-tweets-with-incivil-content, 2021. ↗
  31. Kaggle. turkish offensive language detection. https://www.kaggle.com/datasets/toygarr/turkish-offensive-language-detection, 2022. ↗
  32. Habibe Karayiğit, Çiğdem İnan Acı, and Ali Akdağlı. Detecting abusive instagram comments in turkish using convolutional neural network and machine learning methods. Expert Systems with Applications, 174:114802, 2021. find
  33. Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870, 2023. — [arXiv:2307.07870](https://arxiv.org/abs/2307.07870) find
  34. Jean Lave and Etienne Wenger. Situated learning: Legitimate peripheral participation. Cambridge university press, 1991. find
  35. Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3530– 3538, Gyeongju, Republic of K… find
  36. Joao A Leite, Diego F Silva, Kalina Bontcheva, and Carolina Scarton. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint arXiv:2010.04543, 2020. find
  37. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: Incorporating cultural differences into large language models. arXiv preprint arXiv:2402.10946, 2024.
  38. Lei Li, Yongfeng Zhang, and Li Chen. Prompt distillation for efficient llm-based recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 1348–1357, 2023. find
  39. Margarita Limón. On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and instruction, 11(4-5):357–380, 2001. find
  40. Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André FT Martins, and Hinrich Schütze. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303, 2024.
  41. Yen-Ting Lin and Yun-Nung Chen. Taiwan llm: Bridging the linguistic divide with a culturally aligned language model. arXiv preprint arXiv:2311.17487, 2023. — [arXiv:2311.17487](https://arxiv.org/abs/2311.17487) find
  42. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591, 2023.
  43. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342, 2023. maywell. Synatra. https://huggingface.co/maywell/Synatra-7B-v0.3-dpo… ↗
  44. Jihyung Moon, Won Ik Cho, and Junbum Lee. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25–31, Online, July 2020. Association for Computational Linguistics. URL https:… find
  45. Hamdy Mubarak, Hend Al-Khalifa, and AbdulMohsen Al-Thubaity. Overview of osact5 shared task on arabic offensive language and hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detect…
  46. Cas Mudde. The 2012 stein rokkan lecture: Three decades of popu list radical right parties in western europe: so what? In The Populist Radical Right, pages 545–558. find
  47. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948, 2024.
  48. Tarek Naous, Michael J Ryan, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456, 2023.
  49. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907–1917, 2023a.
  50. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023b.
  51. OpenAI. Chatgpt. https://chat.openai.com/, 2023a. ↗
  52. OpenAI. Gpt-4 technical report, 2023b.
  53. OpenAI. text-embedding-3-small. https://platform.openai.com/docs/guides/embeddings, 2024. ↗
  54. Juan Carlos Pereira-Kohatsu, Lara Quijano-Sánchez, Federico Liberatore, and Miguel Camacho-Collados. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654, 2019. find
  55. Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharnpipitchai. Typhoon: Thai large language models. arXiv preprint arXiv:2312.13951, 2023. — [arXiv:2312.13951](https://arxiv.org/abs/2312.13951) find
  56. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. Sabiá: Portuguese large language models. In Brazilian Conference on Intelligent Systems, pages 226–240. Springer, 2023. find
  57. Flor Miriam Plaza-del Arco, Arturo Montejo-Ráez, L Alfonso Urena Lopez, and María-Teresa Martín-Valdivia. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1096–1108,… find
  58. Abhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310.07251, 2023. find
  59. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464, 2024. — [arXiv:2404.12464](https://arxiv.org/abs/2404.12464) find
  60. Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, and Md Saiful Islam. Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computational Intelligence: IJCACI 2020, pages 457–468. find
  61. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Michael Beißwenger, Michael Wojatzki, and Torsten Zesch, editors, Proceedings of NLP4CMC III: 3rd Wo… find
  62. Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. Multilingual hatecheck: Functional tests for multilingual hate speech detection models. arXiv preprint arXiv:2206.09917, 2022. find
  63. Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of llm alignment on global representation. arXiv preprint arXiv:2402.15018, 2024.
  64. Omar Shaikh, Valentino Chai, Michele J Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309, 2023. find
  65. Omar Sharif and Mohammed Moshiul Hoque. Tackling cyber-aggression: Identification and fine-grained categorization of aggressive texts on social media using weighted ensemble of transformers. Neurocomputing, 490:462–481, 2022. find
  66. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238, 2024.
  67. Hoyun Song, Soo Hyun Ryu, Huije Lee, and Jong C Park. A large-scale comprehensive abusiveness detection dataset with multifaceted labels from reddit. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 552–561, 2021. find
  68. Helen Spencer-Oatey and Peter Franklin. What is culture. A compilation of quotations. GlobalPAD Core Concepts, 1(22):1–21, 2012. find
  69. Pew Global Attitudes Survey. Pew global attitudes survey. https://www.pewresearch.org/, 2022a. ↗
  70. World Values Survey. World values survey. https://www.worldvaluessurvey.org/wvs.jsp, 2022b. ↗
  71. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  72. TurkishSpamV01. Turkish Spam V01. https://doi.org/10.24432/C5WG7F. — [doi:10.24432/c5wg7f](https://doi.org/10.24432/c5wg7f) ↗
  73. UCI Machine Learning Repository, 2019. find
  74. DOI: Francielle Vargas, Isabelle Carvalho, Fabiana Rodrigues de Góes, Thiago Pardo, and Fabrício Benevenuto. HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Co… find
  75. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R Lyu. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprint arXiv:2310.12481, 2023.
  76. Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93, 2016.
  77. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. Overview of the germeval 2018 shared task on the identification of offensive language. find
  78. Verlag der Österreichischen Akademie der Wissenschaften, 2018. find
  79. Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv 2307.09705, 2023.
  80. Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S Bernstein, and John Mitchell. Social skill training with large language models. arXiv preprint arXiv:2404.04204, 2024. yanolja. Eeve. https://huggingface.co/yanolja/EEVE-Korean-10.8B-v1.0, 2024. ↗
  81. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235, 2020. find
  82. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. arXiv preprint arXiv:2404.16308, 2024. find
  83. Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.08011, 2022. — [arXiv:2202.08011](https://arxiv.org/abs/2202.08011) find
  84. Contents A Discussion on the Relationship between Culture and Language find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2024-culturepark-boosting-cross-cultural-understanding/