Elamur Research⌕
☰
home / papers / 2024 / Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys

Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys

Summary

Dialogue systems built on pre-trained language models often carry cultural biases. Few resources exist for making them sensitive to the cultural background of their users. The authors address this with cuDialog, which they describe as the first benchmark for multi-turn dialogue built around culture. They draw on the English version of OpenSubtitles 2018 and keep the 13 most frequent countries of origin that also appear in the Hofstede Culture Survey. They restrict the data to five film genres and reformat it into dialogue histories paired with reference continuations. Each sample is tagged with a country label and that country's six Hofstede dimension scores. Using English only keeps surface language differences out of the way, so models must rely on implicit cultural signals. The benchmark defines three tasks:

  • culture classification
  • regression of cultural dimension scores
  • dialogue generation

For generation, the authors fuse a survey-based cultural vector with the dialogue features inside an encoder-decoder model. This fusion brings outputs closer to the references and improves cultural alignment. Overall, the work shows that cultural attributes can be detected in dialogue and can be used to improve response prediction.

Key points

  • Hofstede dimensions: six scores (pdi, idv, mas, uai, lto, ivr) serve as soft cultural knowledge. Survey results are available for 111 countries and come from a sample of about 117,000 employees.
  • Data splits: splits are made by movie to prevent leakage, with 728/91/104 movies per genre for train/dev/test. The genres are Action, Comedy, Drama, Romance and Crime. Training sets hold roughly 109k–137k samples per genre, and contexts average about 71–72 mT5 tokens.
  • Classification baseline: mBERT identifies American, Australian and Canadian dialogues best, which the authors attribute to English-dominant pre-training. Crime and Action dialogues are the easiest to classify across all cultures.
  • Stated limitations: single-reference evaluation is a weak measure of quality. The English subtitles mix human and machine translation. Equating a language with a culture, or a country with a culture, is a simplification.
  • Availability: the benchmark is released at github.com/yongcaoplus/cuDialog under GPL v3.0, the same license as OpenSubtitles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-eacl.63 · arXiv 2401.10352

@inproceedings{cao2024bridging,
  title = {{Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys}},
  author = {Cao, Yong and Chen, Min and Hershcovich, Daniel},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-eacl.63},
  eprint = {2401.10352},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-eacl.63}
}

References

57 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Faiyaz Al Zamal, Wendy Liu, and Derek Ruths. 2012. Homophily and latent attribute inference: Inferring latent attributes of twitter users from neighbors. In Proceedings of the International AAAI Conference on Web and Social Media, volume 6, pages 387–390. find
  2. Badar Almuhailib. 2019. Analyzing cross-cultural writing differences using contrastive rhetoric: A critical review. Advances in Language and Literary Studies, 10(2):102–106. find
  3. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguis… find
  4. Claire Boine, Michael Siegel, Craig Ross, Eric W Flee-gler, and Ted Alcorn. 2020. What is gun culture? cultural variations and trends across the united states. Humanities and Social Sciences Communications, 7(1):1–12. find
  5. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, Dubrovnik, Croatia. As… find our card
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  7. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Xiao Dong Yue. 2010. Exploration of chinese humor: Historical review, empirical findings, and critical reflections. find
  10. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018a. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne,… find
  11. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018b. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392. Association… find
  12. Kathleen C. Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi’s moral philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 26–42, Seattle, U.S.A. Association for Computational Linguistics… find
  13. Tingchen Fu, Xueliang Zhao, Chongyang Tao, Ji-Rong Wen, and Rui Yan. 2022. There are a thousand hamlets in a thousand people’s eyes: Enhancing knowledge-grounded dialogue with personal memory. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  14. Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437. find
  15. Jia-Chen Gu, Zhenhua Ling, Yu Wu, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2021. Detecting speaker personas from conversational texts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1126–1136, Online and Punta Cana, Dominican Republic. Association fo… find
  16. Jian Guan, Ziqi Liu, and Minlie Huang. 2022. A corpus for understanding and generating moral stories. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5069–5087, Seattle, United States. Associatio… find
  17. Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin A Rothkopf, Alexander Fraser, and Kristian Kersting. 2022. Speaking multiple languages affects the moral bias of language models. arXiv preprint arXiv:2211.07733. find
  18. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi-queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural… find our card
  19. Geert Hofstede. 1984. Culture’s consequences: International differences in work-related values, volume 5. sage. find
  20. Tonglin Jiang, Hao Li, and Yubo Hou. 2019. Cultural differences in humor perception, usage, and implications. Frontiers in psychology, 10:123. find
  21. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785. find
  22. Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Indra Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and multi-cultural figurative language understanding. arXiv preprint arXiv:2305.16171. find
  23. Kamil Kanclerz, Alicja Figas, Marcin Gruza, Tomasz Kajdanowicz, Jan Kocon, Daria Puchalska, and Prze-myslaw Kazienko. 2021. Controversy and conformity: from generalized to personalized aggressiveness detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistic… find
  24. Amr Keleg and Walid Magdy. 2023. Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. arXiv preprint arXiv:2306.05076. find
  25. Wen Lai, Alexandra Chronopoulou, and Alexander Fraser. 2022. m^4 adapter: Multilingual multi-domain adaptation for machine translation with a meta-adapter. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4282–4296, Abu Dhabi, United Arab Emirates. Association for Comp… find
  26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  27. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Reso… find our card
  28. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana,… find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  30. Tim Lomas, Pablo Diego-Rosell, Koichiro Shiba, Priscilla Standridge, Matthew T Lee, Brendan Case, Alden Yuanhong Lai, and Tyler J VanderWeele. 2023. Complexifying individualism versus collectivism and west versus east: Exploring global diversity in perspectives on self and other in the gallup world… find
  31. Weicheng Ma, Samiha Datta, Lili Wang, and Soroush Vosoughi. 2022. EnCBP: A new benchmark dataset for finer-grained cultural background prediction in English. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2811–2823, Dublin, Ireland. Association for Computational Lingui… find
  32. Bodhisattwa Prasad Majumder, Taylor Berg-Kirkpatrick, Julian McAuley, and Harsh Jhamtani. 2021. Unsupervised enrichment of persona-grounded dialog with background stories. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… find
  33. Rod A Martin and Thomas Ford. 2018. The psychology of humor: An integrative approach. Academic press. find
  34. Farhad Moghimifar, Shilin Qu, Tongtong Wu, Yuan-Fang Li, and Gholamreza Haffari. 2023. Normmark: A weakly supervised markov model for socio-cultural norm discovery. arXiv preprint arXiv:2305.16598. find
  35. Richard E Nisbett, Kaiping Peng, Incheol Choi, and Ara Norenzayan. 2001. Culture and systems of thought: holistic versus analytic cognition. Psychological review, 108(2):291. find
  36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  37. Marco Pennacchiotti and Ana-Maria Popescu. 2011. Democrats, republicans and starbucks afficionados: user classification in twitter. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 430– 438. find
  38. Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: A large-scale dataset for personalized chatbot. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, page… find
  39. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. find
  40. Khairiah A Rahman. 2013. Life imitating art: Asian romance movies as a social mirror. Pacific Journalism Review, 19(2):107–121. find
  41. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? find our card
  42. Vered Shwartz. 2022. Good night at 4 pm?! time expressions in different cultures. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2842–2853, Dublin, Ireland. Association for Computational Linguistics. find
  43. Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Cross-cultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti… find our card
  44. Nir Sweed and Dafna Shahaf. 2021. Catchphrase: Automatic detection of cultural references. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1–7, Online… find
  45. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. find
  46. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  47. Mads Toftrup, Søren Asger Sørensen, Manuel R. Ciosici, and Ira Assent. 2021. A reproduction of apple’s bi-directional LSTM models for language identification in short strings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Rese… find
  48. Marc Tomlinson, David Bracewell, and Wayne Krug. 2014. Capturing cultural differences in expressions of intentions. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 48–57, Dublin, Ireland. Dublin City University and Association fo… find
  49. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  50. Deeksha Varshney, Akshara Prabhakar, and Asif Ekbal. 2022. Commonsense and named entity aware knowledge grounded dialogue generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1322–1335, Se… find
  51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  52. Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1956–1970, Online. Association for… find
  53. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  54. Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge, and Pengke Chen. 2022. Mdia: A benchmark for multilingual dialogue generation in 46 languages. arXiv preprint arXiv:2208.13078. — [arXiv:2208.13078](https://arxiv.org/abs/2208.13078) find
  55. Ziqiang Zhang and Junyi Ao. 2022. The YiTrans speech translation system for IWSLT 2022 offline shared task. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 158–168, Dublin, Ireland (in-person and online). Association for Computational Linguistic… find
  56. Li Zhou, Laura Cabello, Yong Cao, and Daniel Hershcovich. 2023a. Cross-cultural transfer learning for Chinese offensive language detection. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 8–15, Dubrovnik, Croatia. Association for Computational Linguistics. find
  57. Power Distance (pdi). It measures the acceptance of unequal power distribution within organizations and institutions. • Individualism (idv). It explores the extent to which individuals are integrated into groups. • Masculinity (mas). It measures individuals’ preference in society for achievement, he… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/cao-2024-bridging-cultural-nuances-dialogue-agents/