Elamur Research⌕
☰
home / papers / 2024 / Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

Paper · 2024 · Yilun Hua and Yoav Artzi

Summary

When people refer to the same things repeatedly, they gradually shorten their descriptions and settle on shared, ad-hoc conventions. This paper asks whether multimodal LLMs show the same drive toward efficiency when the entire conversation stays in their context. The authors build ICCA, an automated evaluation framework based on repeated reference games. In each game, a speaker describes one of four similar images and a listener must identify it, over six rounds. ICCA reuses recorded human-human games from Hawkins et al. (2020b), so no new human studies are needed. Models are tested both as speakers, paired with a GPT4 listener, and as listeners, fed recorded human messages. The framework tracks message length, lexical convergence and selection accuracy. None of the five tested MLLMs spontaneously shortens its own messages. Only the strongest ones do so under heavily engineered prompts, and even then their wording keeps drifting instead of stabilizing. As listeners, models cope better with increasingly terse human messages, and GPT4 is the most human-like. The results expose a gap in current training that matters for cheaper, more natural dialogue systems.

Key points

  • Data and setup: ICCA (In-context Conversational Adaptation) uses 54 human-human interactions from Hawkins et al. (2020b). Each game has 4 images and 6 repetitions, giving 24 trials. ICCA also supports variants, such as showing the image context only once.
  • Models evaluated: The five models tested are IDEFICS, LLaVa-1.5, GPT4-vision, Claude 3 opus, plus Gemini 1.0 Pro Vision.
  • Speaker results: GPT4, Gemini and Claude shorten messages only with explicit, game-specific instructions. They often introduce new words instead of converging on a stable label, so true conventions do not form.
  • Repetition bias: IDEFICS and LLaVa assign higher likelihood to artificial transcripts that repeat the first-round messages than to the real, evolving human transcripts. This held in all 54 interactions.
  • Multi-image weakness and release: IDEFICS loses accuracy as the number of images in its input grows beyond about four, which the authors partly attribute to its Flamingo-style cross-attention. ICCA is released under the MIT license at github.com/lil-lab/ICCA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🎓Cornell TechNew York · United States7 works · 1,320 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDialogue systems · 624 worksVision & language · 657 works

Cite

Cite as Yilun Hua and Yoav Artzi. 2024. Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs. arXiv:2408.01417.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.01417 · arXiv 2408.01417

@misc{hua2024talk,
  title = {{Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs}},
  author = {Yilun Hua and Yoav Artzi},
  year = {2024},
  doi = {10.48550/arxiv.2408.01417},
  eprint = {2408.01417},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.01417}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew B… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. URL https://www.aclweb.org/anthology/D16-1125. ↗
  3. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv, 2024. URL https://arxiv.org/abs/2403.19154. ↗
  4. Anthropic. Introducing the next generation of Claude, 2024. URL https://www.anthropic.com/news/claude-3-family. ↗
  5. Susan E. Brennan and Herbert H. Clark. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1996. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  7. Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. Clembench: Using game play to evaluate chat-optimized language models as conversational agents. arXiv, 2023. URL https://arxiv.org/abs/2305.13455. ↗
  8. Herbert H. Clark. Common ground. Cambridge University Press, 1996. find
  9. Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Perspectives on Socially Shared Cognition. American Psychological Association, 1991. find
  10. Herbert H. Clark and Deanna Wilkes-Gibbs. Referring as a collaborative process. Cognition, 22, 1986. URL https://www.sciencedirect.com/science/article/pii/ 0010027786900107. ↗
  11. Philip R. Cohen and Hector J. Levesque. Rational Interaction as the Basis for Communication. The MIT Press. URL https://doi.org/10.7551/mitpress/3839.003.0014. Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. From rewriting to remembering: Common ground for conver… ↗
  12. Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. Analysis of language change in collaborative instruction following. In Findings of the Association for Computational Linguistics: EMNLP, 2021. URL https://aclanthology.org/2021.findings-emnlp. 239.pdf. ↗
  13. Ron Eliav, Anya Ji, Yoav Artzi, and Robert Hawkins. Semantic uncertainty guides the extension of conventions to new referents. arXiv, 2023. URL http://arxiv.org/abs/2305. 06539. ↗
  14. Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018. URL https://www.aclweb.org/anthology/N18… ↗
  15. Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. How efficiency shapes human language. Trends in Cognitive Sciences, 23, 2019. URL https://www.sciencedirect.com/science/article/pii/ S1364661319300580. ↗
  16. Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20, 2016. URL https://api.semanticscholar.org/ CorpusID:3632786. ↗
  17. Google. Introducing Gemini: Our largest and most capable AI model, 2023. URL https://blog.google/technology/ai/google-gemini-ai/. ↗
  18. Barbara J. Grosz and Candace L. Sidner. Plans for discourse. In Intentions in Communication. The MIT Press. URL https://doi.org/10.7551/mitpress/3839.003.0022. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. The PhotoBook dataset: Building common gro… ↗
  19. Robert Hawkins, Michael C. Frank, and Noah D. Goodman. Characterizing the dynamics of learning in repeated reference games. Cognitive Science, 44, 2020a. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/cogs.12845. ↗
  20. Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. Continual adaptation for efficient machine communication. In Proceedings of the Conference on Computational Natural Language Learning, November 2020b.
  21. William S. Horton and Richard J. Gerrig. Revisiting the memory-based processing approach to common ground. Topics in Cognitive Science, 8, 2016. URL https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12216. — [doi:10.1111/tops.12216](https://doi.org/10.1111/tops.12216) ↗
  22. Huggingface. HuggingFaceM4/idefics-80b-instruct, 2023. URL https://huggingface.co/ HuggingFaceM4/idefics-80b-instruct. ↗
  23. Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, December 2022. URL https://aclanthology.org/2022.emnlp-main.38. ↗
  24. Robert M. Krauss and Sidney Weinheimer. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1, 1964. URL https://link.springer.com/article/10.3758/BF03342817. — [doi:10.3758/bf03342817](https://doi.org/10.3758/bf03342817) ↗
  25. Robert M. Krauss and Sidney Weinheimer. Concurrent feedback, confirmation, and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4, 1966. find
  26. David Kellogg Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969. find
  27. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv, 2023a. URL https://arxiv.org/pdf/2310.03744. ↗
  28. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2023b. URL https://aclanthology.org/ 2024.tacl-1.9.pdf. ↗
  29. Arthur B. Markman and Valerie S. Makin. Referential communication and category acquisition. Journal of Experimental Psychology: General, 127, 1998. find
  30. Bill McDowell and Noah D. Goodman. Learning from omission. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019. URL https://aclanthology.org/P19-1059.pdf. ↗
  31. Charles Metzing and Susan E. Brennan. When conceptual pacts are broken: Partner-specific effects on the comprehension of referring expressions. Journal of Memory and Language, 49, 2003. URL https://www.sciencedirect.com/science/article/pii/S0749596X03000287. ↗
  32. Will Monroe, Robert Hawkins, Noah D. Goodman, and Christopher Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5, 2017. URL https://aclanthology.org/ Q17-1023.pdf. ↗
  33. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — Training language models to follow instructions with human feedback
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014. URL http://www.aclweb.org/anthology/D14-1162. ↗
  36. Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding gaps in language model generations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2024. URL… find
  37. Robert Stalnaker. Common ground. Linguistics and Philosophy, 25, 2002. URL https://link.springer.com/article/10.1023/A:1020867916902. ↗
  38. Alberto Testoni and Raquel Fernández. Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.… ↗
  39. David Traum. A Computational Theory of Grounding in Natural Language Conversation. PhD thesis, University of Rochester, 1994. find
  40. Julia White, Jesse Mu, and Noah D. Goodman. Learning to refer informatively by amortizing pragmatic reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2020. URL https://cognitivesciencesociety.org/cogsci20/papers/0175/0175.pdf. ↗
  41. Kayo Yin, Terry Regier, and Dan Klein. American sign language handshapes reflect pressures for communicative efficiency. arXiv, 2024. URL https://arxiv.org/abs/2406.04024. ↗
  42. Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In Proceedings of the International Conference on Machine Learning, 2021. URL https://proceedings.mlr.press/v139/zhu21d/zhu21d.pdf. ↗
  43. George Kingsley Zipf. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley Press, 1949. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hua-2024-talk-less-interact-better-evaluating/