Elamur Research⌕
☰
home / papers / 2023 / Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

Summary

The paper argues that pragmatic reference, meaning how people work out what a speaker intends, depends on who is talking. A clue that seems obvious to one person can mislead someone with a different cultural background. Earlier reference-game studies mostly treated speakers and listeners as anonymous. To study this effect, the authors built CULTURALCODES, a dataset of English gameplay from an adapted Codenames Duet collected on Amazon Mechanical Turk. Besides game moves and free-text justifications, it records each participant's demographics, personality traits and moral/political values. The authors split a turn into separate prediction problems: picking target words, producing a clue, explaining the clue, guessing, explaining the guess, and detecting pragmatic failure. Each problem was modeled with and without these sociocultural priors. Adding background information consistently helped on both the clue-giver and guesser sides, while a word-vector baseline did poorly. The work shows that sociocultural context can be measured within a controlled game. It also provides a public benchmark for culture-aware pragmatic modeling.

Key points

  • CULTURALCODES contains 794 games and 7,703 turns from 153 unique players (the conclusion gives 156). The data and code are released at github.com/SALT-NLP/codenames.
  • The game board draws on 100 words chosen from the 400-word Codenames Duet pool. These are polysemous nouns (two or more WordNet senses), ranked as most abstract using Brysbaert et al.'s concreteness ratings. Each player's map has 9 goal, 3 avoid and 13 neutral words.
  • Players had to score at least 80% on a rules quiz, were paid $2.50 per game, and games averaged about 17.4 minutes. Free-text rationales were normalized with GPT-3 using few-shot prompting.
  • For pragmatic-failure classification, sociocultural priors increased F-1 by up to +0.10 for RoBERTa and +0.07 for XLNet, both with the Personality features, but only +0.02 for BERT. A fastText nearest-neighbor baseline performed poorly on clue and guess generation.
  • Feeding in all priors together gave the best result only for Target Rationale Generation, which the authors attribute to correlated features causing overfitting. Stated limitations include small Seq2Seq models, a 100-word vocabulary, and evaluation restricted to automatic metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 17 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.410 · arXiv 2306.02475

@inproceedings{shaikh2023modeling,
  title = {{Modeling Cross-Cultural Pragmatic Inference with Codenames Duet}},
  author = {Shaikh, Omar and Ziems, Caleb and Held, William and Pariani, Aryan and Morstatter, Fred and Yang, Diyi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.410},
  eprint = {2306.02475},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.410}
}

References

78 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182, Austin, Texas. Association for Computational Linguistics. ↗ our card
  2. Abhilasha Ashok Kumar, Ketika Garg, and Robert Hawkins. 2021. Contextual flexibility guides communication in a cooperative language game. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43. find
  3. Yuwei Bao, Sayan Ghosh, and Joyce Chai. 2022. Learning to mediate disparities towards pragmatic communication. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2829–2842. find
  4. Basil Bernstein. 2003. Class, codes and control: Applied studies towards a sociology of language, volume 2. Psychology Press. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. find
  6. Baxter S Eaves Jr, Naomi H Feldman, Thomas L Griffiths, and Patrick Shafto. 2016. Infant-directed speech is consistent with teaching. Psychological review, 123(6):758. ↗
  7. Penelope Eckert and Sally McConnell-Ginet. 2013. Language and gender. Cambridge University Press. ↗
  8. Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074. find
  9. Susan T Fiske and Martha G Cox. 1979. Person concepts: The effect of target familiarity and descriptive purpose on the process of describing others 1. Journal of Personality, 47(1):136–161. find
  10. Michael Franke and Gerhard Jäger. 2016. Probabilistic pragmatics, or why bayes’ rule is probably important for pragmatics. Zeitschrift für sprachwissenschaft, 35(1):3–44. ↗
  11. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013. ↗ our card
  12. Geert H Hofstede. 2001. Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations. sage. find
  13. Liye Fu, Susan Fussell, and Cristian Danescu-Niculescu-Mizil. 2020. Facilitating the communication of politeness through fine-grained paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5127–5140, Online. Association for Computational Linguistics. find our card
  14. Samee Ibraheem, Gaoyue Zhou, and John DeNero. 2022. Putting the con in context: Identifying deceptive actors in the game of mafia. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 158–168, Seattle, United States. Association for Computational Linguistics. find
  15. Aparna Garimella, Carmen Banea, and Rada Mihalcea. 2017. Demographic-aware word associations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2285–2295, Copenhagen, Denmark. Association for Computational Linguistics. find
  16. Julian Jara-Ettinger, Hyowon Gweon, Laura E Schulz, and Joshua B Tenenbaum. 2016. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8):589–604. ↗
  17. Noah D Goodman and Michael C Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829. ↗
  18. Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Koleva Spassena, and Peter H Ditto. 2008. Moral foundations questionnaire. Journal of Personality and Social Psychology. find
  19. Lisa J Green. 2002. African American English: a linguistic introduction. Cambridge University Press. ↗
  20. Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. 2008. Bayesian models of cognition. find
  21. William B Gudykunst and Young Yun Kim. 1984. Communicating with strangers: An approach to intercultural communication. Addison Wesley Publishing Company. find
  22. Jonathan Haidt. 2012. The righteous mind: Why good people are divided by politics and religion. Vintage. find
  23. Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. Social Justice Research, 20(1):98–116. ↗
  24. Evelyn Hatch et al. 1992. Discourse and language education. Cambridge University Press. find
  25. Robert XD Hawkins, Andreas Stuhlmüller, Judith Degen, and Noah D Goodman. 2015. Why do you ask? good questions provoke informative answers. In CogSci. find
  26. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi-queras, Ilias Chalkidis, Ruixiang Cui, et al. 2022. Challenges and strategies in cross-cultural nlp. In Catalina Jaramillo, Megan Charity, Rodrigo Canaan, and Julian Togelius. 2020. Word autobots: Using transformers for word association in the game codenames. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 231–237. ↗ our card
  27. Alan Jern, Christopher G Lucas, and Charles Kemp. 2017. People learn other people’s preferences through inverse decision-making. Cognition, 168:46– 64. ↗
  28. Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big five inventory. Journal of Personality and Social Psychology. find
  29. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016. How do cultural differences impact the quality of sarcasm annotation?: A case study of Indian annotators and American text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 95–99, Berlin, Germany. Association for Computational Linguistics. find our card
  30. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 262–272, Valencia, Spain. Association for Computational Linguistics. find our card
  31. Fereshte Khani, Noah D. Goodman, and Percy Liang. 2018. Planning, inference and pragmatics in sequential language games. Transactions of the Association for Computational Linguistics, 6:543–555. ↗ our card
  32. Andrew Kim, Maxim Ruzmaykin, Aaron Truong, and Adam Summerville. 2019. Cooperation and codenames: Understanding natural language processing via codenames. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 160–166. find
  33. Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will I sound like me? improving persona consistency in dialogues through pragmatic self-consciousness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 904–916, Online. Association for Computational Linguistics. ↗ our card
  34. Alena Korshuk. 2007. Learning more about cultures through free word association data. find
  35. Collin J Kovacs, Jasper M Wilson, and Abhilasha A Kumar. 2022. Fast and frugal memory search for communication. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44. ↗
  36. Divya Koyyalagunta, Anna Sun, Rachel Lea Draelos, and Cynthia Rudin. 2021. Playing codenames with language graphs and word embeddings. Journal of Artificial Intelligence Research, 71:319–346. ↗
  37. Abhilasha A Kumar, Mark Steyvers, and David A Balota. 2021. Semantic memory search and retrieval in a novel cooperative word game: A comparison of associative and distributional semantic models. Cognitive Science, 45(10):e13053. ↗
  38. William Labov. 2011. Principles of linguistic change, volume 3: Cognitive and cultural factors, volume 3. John Wiley & Sons. find
  39. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗ our card
  40. Angeline Lillard. 1998. Ethnopsychologies: cultural variations in theories of mind. Psychological bulletin, 123(1):3. ↗
  41. Angeline Lillard. 1999. Developing a cultural theory of mind: The ciao approach. Current Directions in Psychological Science, 8(2):57–61. ↗
  42. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  43. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692. ↗
  44. Kate Loveys, Jonathan Torrez, Alex Fine, Glen Mori-arty, and Glen Coppersmith. 2018. Cross-cultural differences in language markers of depression online. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 78–87, New Orleans, LA. Association for Computational Linguistics. find
  45. George A Miller. 1974. Psychology, language, and levels of communication. In Human communication. John Wiley. find
  46. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. ↗ our card
  47. Joan G Miller. 1984. Culture and the development of everyday social explanation. Journal of personality and social psychology, 46(5):961. ↗
  48. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. ↗ our card
  49. Aaron J Moss, Cheskie Rosenzweig, Jonathan Robinson, and Leib Litman. 2020. Demographic stability on mechanical turk despite covid-19. Trends in cognitive sciences, 24(9):678–680. ↗
  50. Ira Noveck. 2018. Experimental pragmatics: The making of a cognitive science. Cambridge University Press. find
  51. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  52. Christopher Potts. 2012. Goal-driven answers in the cards dialogue corpus. In Proceedings of the 30th West Coast Conference on Formal Linguistics, pages 1–20. Cascadilla Proceedings Project. find
  53. James E Purpura. 2004. Assessing grammar, volume 8. Cambridge University Press. find
  54. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. ↗
  55. Yael Ravin and Claudia Leacock. 2000. Polysemy: Theoretical and computational approaches. OUP Oxford. find
  56. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics. find our card
  57. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. ArXiv preprint, abs/2210.13312. ↗
  58. Raheleh Saryazdi, Joanne Nuque, and Craig G Chambers. 2022. Pragmatic inferences in aging and human-robot communication. Cognition, 223:105017. ↗
  59. Wilbur Schramm. 1954. How communication works. The process and effects of mass communication, 3:26. find
  60. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609. ↗
  61. Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019. Pragmatically informative text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4060–4067, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  62. Blum-Kulka Shoshana, Juliane House, and Gabriele Kasper. 1989. Cross-cultural pragmatics: Requests and apologies. Grazer Linguistische Studien. find
  63. Richard A Shweder. 1984. Anthropology’s romantic rebellion against the enlightenment, or there’s more to thinking than reason and evidence. Culture theory: Essays on mind, self, and emotion, pages 27–66. find
  64. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan-neershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489. find
  65. Felix Soldner, Verónica Pérez-Rosas, and Rada Mihalcea. 2019. Box of lies: Multimodal deception detection in dialogues. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1768–1777, Minneapolis, Minnesota. Association for Computational Linguistics. find
  66. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 543–553, Berlin, Germany. Association for Computational Linguistics. find our card
  67. Darcy Sperlich, Jaiho Leem, and Eui-Jeen Ahn. 2016. The interaction of politeness systems in Korean learners of French. In Proceedings of the 30th Pacific Asia Conference on Language, Information and Computation: Oral Papers, pages 163–171, Seoul, South Korea. find
  68. Stefanie Stadler. 2012. Cross-cultural pragmatics. The Encyclopedia of applied linguistics, pages 1–8. ↗
  69. Naoko Taguchi. 2012. Context, individual differences and pragmatic competence. In Context, Individual Differences and Pragmatic Competence. Multilingual Matters. ↗
  70. Jenny Thomas. 1983. Cross-cultural pragmatic failure. Applied linguistics, 4(2):91–112. ↗
  71. EV Tikhonova. 2014. Linguistic diagnosing of religious relationships through word association responses. In Conference proceedings of international multidisciplinary scientific conference on social sciences and arts, volume 3, pages 505–516. find
  72. Chvátil Vlaada. 2015. Codenames – rules - czech games edition | boardgame publisher. find
  73. Phillip Wolff and Kevin J Holmes. 2011. Linguistic relativity. Wiley Interdisciplinary Reviews: Cognitive Science, 2(3):253–265. find
  74. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754–5764. ↗ our card
  75. Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. 2021. Adversarial language games for advanced natural language intelligence. In Proceedings of AAAI. find
  76. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe-view.net. ↗
  77. Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris Callison-Burch, Yejin Choi, and Prithvi-raj Ammanabrolu. 2022. An ai dungeon master’s guide: Learning to converse and guide with intents and theory-of-mind in dungeons and dragons. ArXiv preprint, abs/2212.10060. ↗
  78. Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755–3773. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/shaikh-2023-modeling-cross-cultural-pragmatic-inference/