Elamur Research⌕
☰
home / papers / 2024 / Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames

Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames

Paper · 2024 · Isadora White, Sashrika Pandey and Michelle Pan

Summary

Speakers of the same language can still misunderstand each other when their background knowledge differs. The authors' example is "boot", which means a car's trunk to a British speaker and a shoe to an American one. This paper asks whether an artificial clue giver can detect such a gap during play and adapt to it. The authors propose RSA+C3, an extension of the Rational Speech Acts model. The pragmatic speaker keeps several literal-listener models, one per cultural group, and estimates which group best explains the partner's actual guesses. It then chooses clues tuned to that group. The testbed is the cooperative word game Codenames Duet together with the Cultural Codes dataset. Simulated givers and guessers are built from contrastively trained word embeddings and from few-shot LLM prompting. On a fixed set of 100 boards, RSA+C3 givers win more often than plain RSA or literal givers, against both embedding-based and Llama2-7B-Chat guessers. The work matters because it treats common ground as something to infer turn by turn rather than take for granted.

Key points

  • Data: the Cultural Codes dataset (Shaikh et al., 2023) has 794 Codenames Duet games from 153 players, with demographic surveys. It is split 80/10/10, and the players in the validation/test sets do not appear in training.
  • Game setup: each board has 25 words, of which 9 are goal words, 3 are avoid words (an instant loss) and 13 are neutral words (they end the turn).
  • Human alignment is measured with four metrics: giver target accuracy, clue accuracy, guess accuracy and guesser target accuracy. On these, the embedding-based players beat few-shot prompted LLMs.
  • RSA+C3 sets each culture's weight to the probability that its literal listener would produce the observed guess given the clue. The speaker then plans with the highest-scoring listener. The tuned settings are α = 0.5 and a neutral penalty of 0.1, with standard error computed over three runs.
  • Cultures are defined mainly by education level (graduate, bachelor, high school). The dataset is skewed, with 78% White and 58% liberal participants, which mutes some cultural differences.

Interesting ideas

  • I1. Instead of a single model of the listener, the speaker carries a small bank of candidate listener models and picks one by how well it predicts the partner's observed reactions; this is a cheap, transparent form of partner modelling that applies to any dialogue.
  • I2. The clue's cost is defined as the chance the listener lands on a harmful or useless option, so the speaker chooses wording by its risk of misreading as well as by its informativeness.
  • I3. All methods are compared on one frozen set of 100 boards, so a higher win rate cannot come from luck in board difficulty.
  • I4. Simulated players are first checked against human behaviour with alignment metrics, and only then used as partners in interaction experiments; an LLM-built guesser stands in for an unknown, differently constructed partner.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): implicature and speech-act interpretation in prag could keep a few reader/speaker profiles (e.g. UK vs US lexical senses) and choose the one that best explains earlier turns, logging the choice as an explanation.
  • I2 (back): generation could parse its own candidate sentences and penalise those whose recovered frames include wrong or dangerous readings, preferring the least ambiguous wording.
  • I3 (learn): self-play and repair-rule experiments should be scored on a fixed, versioned item set, so improvements are not artefacts of easier samples.
  • I4 (learn): LLM teacher tiers and simulated students could be validated against human answers before use, with a differently built model kept as an out-of-distribution test partner.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Isadora White, Sashrika Pandey and Michelle Pan. 2024. Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames. arXiv:2408.04900.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.04900 · arXiv 2408.04900

@misc{white2024communicate,
  title = {{Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames}},
  author = {Isadora White, Sashrika Pandey and Michelle Pan},
  year = {2024},
  doi = {10.48550/arxiv.2408.04900},
  eprint = {2408.04900},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.04900}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling" culture" in llms: A survey. arXiv preprint arXiv:2403.15412. ↗
  2. Nouhoum Kone. 2020. Speech acts in un treaties: A pragmatic perspective. Open Journal of Modern Linguistics, 10(6):813–827. ↗
  3. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery. find
  4. Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajizhirzi, and Yejin Choi. 2022. Aligning to social norms and values in interactive narratives. In North American Chapter of the Association for Computational Linguistics (NAACL). ↗
  5. Divya Koyyalagunta, Anna Y. Sun, Rachel Lea Draelos, and Cynthia Rudin. 2021. Playing codenames with language graphs and word embeddings. CoRR, abs/2105.05885. ↗
  6. Carolyn Jane Anderson and Brian W. Dillon. 2019. Guess who’s coming (and who’s going): Bringing perspective to the rational speech acts framework. Proceedings of the Society for Computation in Linguistics, 2(20):185–194. ↗
  7. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. Culturellm: Incorporating cultural differences into large language models. arXiv preprint arXiv:2402.10946. ↗
  8. Joseph Bills and Christopher Archibald. 2023. A deductive agent hierarchy: Strategic reasoning in codenames. In 2023 IEEE Conference on Games (CoG), pages 1–8. find
  9. Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024b. Culturepark: Boosting cross-cultural understanding in large language models. arXiv preprint arXiv:2405.15145. ↗
  10. Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. Culturalteaming: Ai-assisted interactive red-teaming for challenging llms’(lack of) multicultural knowledge. arXiv preprint arXiv:2404.06664. ↗
  11. Eleonore Lumer and Hendrik Buschmeier. 2022. Modeling social influences on indirectness in a rational speech act approach to politeness. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44. ↗
  12. Judith Degen. 2023. The rational speech act framework. Annual Review of Linguistics, 9:519–540. ↗
  13. Paweł Niszczota and Mateusz Janczak. 2023. Large language models can replicate cross-cultural differences in personality. arXiv preprint arXiv:2310.10679. ↗
  14. Judith Degen, Michael Henry Tessler, and Noah D Goodman. 2015. Wonky worlds: Listeners revise world knowledge when utterances are odd. In CogSci. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. For all of the following experiments, we used default hyperparameters as provided in the open-source Llama2 code 2 and model sizes of 7B and 13B. The following experiments were conducted over the validation set of the Cultural Codes dataset. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. ICML. find
  15. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. A.1 Clue generation We prompted the 7B and 13B Llama2-Chat models to generate clues using the following few-shot prompt, allowing for a flexible free-form text generation informed by prior examples of a Codenames-style clue: Martin J Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(2):169–190.
  16. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/white-2024-communicate-play-pragmatic-reasoning-efficient-cross/