Elamur Research⌕
☰
home / papers / 2026 / LVLMs and Humans Ground Differently in Referential Communication

LVLMs and Humans Ground Differently in Referential Communication

Paper · 2026 · Peter Zeng, Weiling Li, Amie J. Paige et al.

Summary

The paper asks whether large vision-language models (LVLMs) can build common ground with a partner the way people do when they repeatedly refer to hard-to-name objects. The authors ran a live, multi-turn director–matcher picture-matching game modelled on Clark and Wilkes-Gibbs (1986). The stimuli were baskets with no obvious conventional labels. The design was fully factorial: human-human, human-AI, AI-human and AI-AI pairs, with GPT-5.2 as the model, played over four rounds. Pairs were scored on accuracy, efficiency (words and turns) and lexical overlap.

Only human-human pairs became more accurate across rounds while also using less effort. This pattern reflects entrainment on short, shared referring expressions, known as conceptual pacts. GPT-5.2 showed no sign of accumulating common ground, even when paired with itself and given the full dialogue history. AI-AI accuracy dropped over rounds while effort stayed flat. Pairs with an AI director performed especially poorly. The findings point to real risks when AI agents must take the initiative in collaborative, human-facing tasks.

Key points

  • The released corpus has 356 dialogues (89 pairs × 4 rounds). It comes with the online data-collection pipeline and analysis tools for accuracy, efficiency and lexical overlap: github.com/peterzeng/lvlms-referential-game.
  • The authors describe this as the first study to cover all four human/AI combinations in asymmetric director/matcher roles with real-time multi-turn interaction.
  • Human-human pairs alone improved significantly in accuracy across rounds. AI-AI pairs (GPT-5.2) declined significantly, and accuracy collapsed when the AI was the director of a human matcher.
  • Participants were recruited via Prolific at $12/hour. The study was English-only, used a single object type, and ran the full design with one LVLM, adding limited follow-ups with other proprietary models but no open-weight models.
  • Lexical entrainment was measured automatically as a proxy for human coding and likely underestimates it. The authors argue that prompt engineering is unlikely to fix human-AI grounding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States43 works · 2,571 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

Cite

Cite as Peter Zeng, Weiling Li, Amie J. Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelinsky, Susan E. Brennan and Owen Rambow. 2026. LVLMs and Humans Ground Differently in Referential Communication. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.410

@inproceedings{zeng2026lvlms,
  title = {{LVLMs and Humans Ground Differently in Referential Communication}},
  author = {Peter Zeng, Weiling Li, Amie J. Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelinsky, Susan E. Brennan and Owen Rambow},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.410},
  url = {https://doi.org/10.18653/v1/2026.acl-long.410}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude 4.5 opus. https://www.anthropic.com/claude. Version: 20251101, Accessed: 2026-01-01. ↗
  2. Susan E Brennan and Herbert H Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of experimental psychology: Learning, memory, and cognition, 22(6):1482. find
  3. Susan E Brennan and Eric A Hulteen. 1995. Interaction and feedback in a spoken language system: A theoretical framework. Knowledge-based systems, 8(2-3):143–151. find
  4. Daniel L Chen, Martin Schonger, and Chris Wickens. 2016. oTree—an open-source platform for laboratory, online, and field experiments. Journal of Behavioral and Experimental Finance, 9:88–97. find
  5. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors, Perspectives on socially shared cognition, pages 127–149. APA, Washington, DC. Reprinted in R. M. Baecker (Ed.), Groupware and computer-supported cooperat… find
  6. San Mateo, CA: Morgan Kaufman Publishers, Inc., 1994. find
  7. Herbert H. Clark and Catherine R. Marshall. 1981. Definite knowledge and mutual knowledge. In Aravind Joshi, Bonnie Webber, and Ivan Sag, editors, Elements of Discourse Understanding, pages 10–63. Cambridge University Press, Cambridge, UK. find
  8. Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. find
  9. S. Garrod and A. Anderson. 1987. Saying what you mean in dialogue: A study in conceptual and semantic co-ordination. Cognition, 27(2):181–218. find
  10. Google. 2025. Gemini 3 pro. https://deepmind.google/technologies/gemini/. Accessed: 2026-01-01. ↗
  11. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and semantics 3: Speech acts, pages 41–58. Academic Press, New York. From the William James lectures, Harvard University, 1967. find
  12. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910, Florenc…
  13. Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. 2020. Continual adaptation for efficient machine communication. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 408–419, Online. Association for Computational Linguistics.
  14. Yilun Hua and Yoav Artzi. 2024. Talk less, interact better: Evaluating in-context conversational adaptation in multimodal LLMs. In First Conference on Language Modeling.
  15. Yilun Hua, Evan Wang, and Yoav Artzi. 2025. Post-training for efficient communication via convention formation. arXiv preprint arXiv:2508.06482. — [arXiv:2508.06482](https://arxiv.org/abs/2508.06482) find
  16. Saki Imai, Mert İnan, Anthony Sicilia, and Malihe Alikhani. 2025. Measuring how (not just whether) VLMs build common ground. arXiv preprint arXiv:2509.03805. find
  17. Gail Jefferson. 1989. Preliminary notes on a possible metric which provides for a ‘standard maximum’ silence of approximately one second in conversation. In Derek Roger and Peter Bull, editors, Conversation: An Interdisciplinary Perspective, chapter 8, page 166–196. Multilingual Matters, Clevedon, E… find
  18. Cameron R Jones, Agnese Lombardi, Kyle Mahowald, and Benjamin K Bergen. 2026. LLMs and people both learn to form conventions–just not with each other. arXiv preprint arXiv:2602.08208. — [arXiv:2602.08208](https://arxiv.org/abs/2602.08208) find
  19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  21. C. B. Lockridge and S. E. Brennan. 2001. [Unpublished corpus]. Department of Psychology, Stony Brook University. find
  22. OpenAI. 2025. Gpt-5.2. https://openai.com/ gpt-5. Accessed: 2026-01-01. ↗
  23. Prolific. 2014. Prolific: High-quality human data for AI and academic research. Online platform. find
  24. Version: December 2025. find
  25. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  26. Rupak Sarkar, Neha Srikanth, Taylor Pellegrin, Rachel Rudinger, Claire Bonial, and Philip Resnik. 2025. Understanding common ground misalignment in (b) whether they believed their partner was AI (scale and free response) (c) their personal AI use (multiple choice) goal-oriented dialog: A case-study… find
  27. Michael F Schober and Herbert H Clark. 1989. Understanding by addressees and overhearers. Cognitive Psychology, 21(2):211–232. find
  28. Omar Shaikh, Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2025. Navigating rifts in human-LLM grounding: Study and benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20832–20847, Vienna, Austria. Ass…
  29. Zineng Tang, Lingjun Mao, and Alane Suhr. 2024. Grounding language in multi-perspective referential communication. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19727–19741, Miami, Florida, USA. Association for Computational Linguistics. find
  30. David R. Traum and James F. Allen. 1992. A "speech acts" approach to grounding in conversation. In IC-SLP. find
  31. Zhengxiang Wang, Weiling Li, Panagiotis Kaliosis, Owen Rambow, and Susan Brennan. 2025. LVLMs are bad at overhearing human referential communication. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16769–16793, Suzhou, China. Association for Computati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-lvlms-humans-ground-differently-referential/