Elamur Research⌕
☰
home / papers / 2023 / Multilingual Language Models are not Multicultural: A Case Study in Emotion

Multilingual Language Models are not Multicultural: A Case Study in Emotion

Paper · 2023 · Shreya Havaldar, Sunny Rai, Bhumika Singhal et al.

Summary

Multilingual LMs are used in emotionally sensitive applications such as therapy bots and writing assistants. Yet emotions are felt and put into words differently across cultures. The authors ask whether popular 2023 multilingual models capture this variation in English, Spanish, Chinese (Simplified) and Japanese.

For representations, they compare emotion-phrase embeddings from monolingual RoBERTa models, XLM-RoBERTa, and an explicitly aligned XLM-RoBERTa variant. Their distance-based similarity measure correlates pairwise distances between emotion phrases across two models, so the embedding spaces never need to be aligned directly. They also project embeddings onto the Valence-Arousal plane and test GPT-3 probabilities against documented US–Japan contrasts in Pride and Shame.

For generation, GPT-3.5 and GPT-4 answer culturally loaded scenarios. Culture is signaled either by an English statement or by prompting in the native language, and native speakers rate the replies.

Embeddings prove anchored to English, and generated answers follow Western norms, particularly for Eastern languages. The study cautions against treating emotion labels as translation-invariant and provides a reusable baseline for auditing cultural awareness.

Key points

  • Four high-resource languages were studied, two Western (English, Spanish) and two Eastern (Chinese, Japanese). Each emotion was represented by averaging "I feel …" and "I am …" phrases, machine-translated and spot-checked by native speakers.
  • The embedding analysis covered 271 emotion terms (Davis, 2023). It compared monolingual RoBERTa, XLM-RoBERTa, and a distillation-aligned XLM-RoBERTa (Reimers and Gurevych, 2020), using Pearson correlation of inter-emotion distance vectors.
  • Known cultural contrasts in Pride and Shame between the US and Japan (Tsai et al., 2006) were not reliably reflected by embeddings or by GPT-3 probabilities.
  • A user study had eight volunteers, in four language pairs, rate GPT-3.5 and GPT-4 outputs on a 1–7 cultural-awareness scale. English replies scored highest. Chinese and Japanese replies were worse when the prompt itself was in the native language.
  • Code is released at github.com/shreyahavaldar/Multicultural_Emotion. Stated limitations include a small annotator pool, no low-resource languages, and treating each language as a single culture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu, Sharath Chandra Guntuku and Lyle Ungar. 2023. Multilingual Language Models are not Multicultural: A Case Study in Emotion. Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.wassa-1.19

@inproceedings{havaldar2023multilingual,
  title = {{Multilingual Language Models are not Multicultural: A Case Study in Emotion}},
  author = {Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu, Sharath Chandra Guntuku and Lyle Ungar},
  booktitle = {Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, \& Social Media Analysis},
  year = {2023},
  doi = {10.18653/v1/2023.wassa-1.19},
  url = {https://doi.org/10.18653/v1/2023.wassa-1.19}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguis…
  2. Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. XLM-EMO: Multilingual emotion prediction in social media text. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 195–203, Dublin, Ireland. Association for Computational Lingui… find
  3. Sven Buechel, Susanna Rücker, and Udo Hahn. 2020. Learning and evaluating emotion lexicons for 91 languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1202–1217, Online. Association for Computational Linguistics. find
  4. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study.
  5. Tianyu Cho and Kei Sawada. 2021. Pre-learning model for japanese natural language processing. Japanese Society for Artificial Intelligence Research Group Material Language/Speech Understanding and Dialogue Processing, 93:169–170. find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
  7. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657–668, Online. Association for Computat…
  8. Tchiki Davis. 2023. List of emotions: 271 emotion words. find
  9. Javier De la Rosa, Eduardo G. Ponferrada, Manu Romero, Paulo Villegas, Pablo González de Prado Salas, and María Grandury. 2022. Bertin: Efficient pre-training of a spanish language model using perplexity sampling. Procesamiento del Lenguaje Natural, 68(0):13–23. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  11. Paul Ekman et al. 1999. Basic emotions. Handbook of cognition and emotion, 98(45-60):16. find
  12. Emi Furukawa, June Tangney, and Fumiko Higashibara. 2012. Cross-cultural continuities and discontinuities in shame, guilt, and pride: A study of children residing in japan, korea and the usa. Self and Identity, 11(1):90–113. find
  13. Fred Genesee. 1982. The social psychological significance of code switching in cross-cultural communication. Journal of language and social psychology, 1(1):1–27. find
  14. Elizabeth A Kensinger and Daniel L Schacter. 2006. Processing emotional pictures and words: Effects of valence and arousal. Cognitive, Affective, & Behavioral Neuroscience, 6(2):110–126. find
  15. Shinobu Kitayama, Hazel Rose Markus, and Masaru Kurokawa. 2000. Culture, emotion, and well-being: Good feelings in japan and the united states. Cognition & Emotion, 14(1):93–124. find
  16. Michael Lewis, Kiyoko Takai-Kawakami, Kiyobumi Kawakami, and Margaret Wolan Sullivan. 2010. Cultural differences in emotional responses to success and failure. International journal of behavioral development, 34(1):53–61. find
  17. Nangyeon Lim. 2016. Cultural differences in emotion: differences in emotional arousal level between the east and the west. Integrative Medicine Research, 5(2):105–109. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  19. Gabriel Magno and Virgilio Almeida. 2021. Measuring international online human values with word embeddings. ACM Trans. Web, 16(2). find
  20. Batja Mesquita. 2022. Between us: How cultures create emotions. WW Norton & Company. find
  21. Batja Mesquita, Nico H Frijda, and Klaus R Scherer. 1997. Culture and emotion. Handbook of cross-cultural psychology, 2:255–297. find
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  23. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  24. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161. find
  25. James A Russell, Maria Lewicka, and Toomas Niit. 1989. A cross-cultural study of a circumplex model of affect. Journal of personality and social psychology, 57(5):848. find
  26. Hanieh Shakeri, Carman Neustaedter, and Steve Di-Paola. 2021. Saga: Collaborative storytelling with gpt-3. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’21, page 163–166, New York, NY, USA. Association for Computing Machinery. find
  27. Rene M Stulz and Rohan Williamson. 2003. Culture, openness, and finance. Journal of financial Economics, 70(3):313–349. find
  28. Jonas Thiergart, Stefan Huber, and Thomas Übellacker. 2021. Understanding emails and drafting responses -an approach using GPT-3. CoRR, abs/2102.03062. — [arXiv:2102.03062](https://arxiv.org/abs/2102.03062) find
  29. Jeanne L Tsai. 2017. Ideal affect in daily life: Implications for affective experience, health, and social behavior. Current Opinion in Psychology, 17:118– 128. find
  30. Jeanne L Tsai, Robert W Levenson, and Kimberly Mc-Coy. 2006. Cultural and temperamental variation in emotional response. Emotion, 6(3):484. find
  31. Lu Wang, Munif Ishad Mujib, Jake Ryland Williams, George Demiris, and Jina Huh-Yoo. 2021. An evaluation of generative pre-training model-based therapy chatbot for caregivers. CoRR, abs/2107.13115. — [arXiv:2107.13115](https://arxiv.org/abs/2107.13115) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/havaldar-2023-multilingual-language-models-are-not/