Elamur Research⌕
☰
home / papers / 2023 / Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations

Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations

Summary

Successful conversation depends on common ground (CG): the beliefs that each participant assumes are mutually shared. Cognitive science has studied CG extensively, but NLP has paid it little attention. The authors present what they describe as the first corpus annotated for CG. It is built on unscripted two-party phone dialogs between close friends and relatives from LDC CALLHOME American Speech.

The annotation has two stages:

  • Event extraction: annotators turn clauses into explicit propositions, resolving pronouns and adding speech-act and negation events where needed.
  • Belief and CG labelling: each speaker's belief about each event is marked, and then CG is labelled separately for each speaker. This makes it possible to record cases where the two speakers' views diverge or they misunderstand each other.

The authors also report baseline experiments on extracting events, predicting beliefs and predicting CG updates, including FLAN-T5 models given varying amounts of dialog context. Predicting CG without gold belief labels proved difficult, largely because of class imbalance. The resource matters because it supplies naturalistic data for modelling Theory-of-Mind-like reasoning, an area where current LLMs remain fragile.

Key points

  • The corpus is layered on LDC CALLHOME American Speech (Canavan et al., 1997), a collection of 120 unscripted dialogs between people who know each other well.
  • Belief labels are a simplified version of the FactBank scheme: CT+ (certainly true), CT- (certainly not true), PS (possibly true) and NB (no belief).
  • CG update labels are assigned per speaker:
    • JA: newly and jointly accepted into CG.
    • IN: already part of CG.
    • RT: considered but rejected by the addressee.
  • In error analysis of FLAN-T5 with a window size of 8, the model made 29 errors on 311 events. Of these, 27 were IN and JA confused with each other (in a 13:14 split).
  • Planned extensions include adding audio and intonation, using existing belief taggers and multi-task learning, trying graph neural networks, and annotating higher-order beliefs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.551 · arXiv 2311.01273

@inproceedings{markowska2023finding,
  title = {{Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations}},
  author = {Markowska, Magdalena and Taghizadeh, Mohammad and Soubki, Adil and Mirroshandel, Seyed and Rambow, Owen},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.551},
  eprint = {2311.01273},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.551}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. 1985. Does the autistic child have a “theory of Josep Call and Michael Tomasello. 1998. Distinguishing intentional from accidental actions in orangutans (pongo pygmaeus), chimpanzees (pan troglodytes), and human children (homo sapiens). Journal of co… find
  2. Alexandra Canavan, David Graff, and George Zipperlen. 1997. CALLHOME American English Speech LDC97S42. Web Download. Philadelphia: Linguistic Data Consortium. find
  3. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  4. Herbert H. Clark. 1996. Common ground, ’Using’ Linguistic Books, page 92–122. Cambridge University Press. find
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  6. Philip R. Cohen and Hector J. Levesque. 1990. Rational interaction as the basis for communication. In Jerry Morgan Philip Cohen and James Allen, editors, Intentions in Communication. MIT Press, Cambridge, MA. find
  7. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107–124. find our card
  8. Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2022. From rewriting to remembering: Common ground for conversational qa models. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  10. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 68–73. find
  11. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  12. Barbara J. Grosz and Candace Sidner. 1990. Plans for discourse. In Intentions in Communication. MIT Press, Cambridge, MA. find
  13. William S. Horton and Richard J. Gerrig. 2016. Revisiting the memory-based processing approach to common ground. Topics in Cognitive Science, 8(4):780– 795. find
  14. Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails on pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097. find our card
  15. Lauri Karttunen. 1971a. Implicative verbs. Language, pages 340–358. find
  16. Lauri Karttunen. 1971b. Some observations on factivity. Research on Language & Social Interaction, 4:55– 69. find
  17. Paul Kiparsky and Carol Kiparsky. 1968. Fact. Linguistics Club, Indiana University. find
  18. Michal Kosinski. 2023. Theory of mind might have spontaneously emerged in large language models. find
  19. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–74. find
  20. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal. Association for Computational Linguisti… find
  21. David Lewis. 1969. Convention: A philosophical study. Cambridge MA: Harvard University Press. find
  22. John Murzaku, Tyler Osborne, Amittai Aviram, and Owen Rambow. 2023. Towards generative event factuality prediction. In Findings of the Association for Computational Linguistics: ACL 2023, pages 701– 715, Toronto, Canada. Association for Computational Linguistics. find
  23. John Murzaku, Peter Zeng, Magdalena Markowska, and Owen Rambow. 2022. Re-examining factbank: Predicting the author’s presentation of factuality. In Proceedings of the 29th International Conference on Computational Linguistics, pages 786–796. find
  24. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of the fifth international workshop on inference in computational semantics (icos-5). find our card
  25. Josef Perner, Susan R Leekam, and Heinz Wimmer. 1987. Three-year-olds’ difficulty with false belief: The case for a conceptual deficit. British journal of developmental psychology, 5(2):125–137. find
  26. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4393–4399, Florence, Italy. Associati… find our card
  27. Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg, Owen Rambow, Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gregory Werner, Yorick Wilks, and Janyce Wiebe. 2015. A new dat… find
  28. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China. find
  29. Coling 2010 Organizing Committee. find
  30. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526. find
  31. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  32. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China… find our card
  33. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744, New Orleans, Louis… find our card
  34. Roser Saurí and James Pustejovsky. 2009. FactBank: a corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. find
  35. Damien Sileo and Antoine Lernould. 2023. Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic. find
  36. Robert C. Stalnaker. 2002. Common ground. Linguistics and Philosophy, 25(5-6):701–721. find
  37. Jennifer Tracey, Owen Rambow, Claire Cardie, Adam Dalton, Hoa Trang Dang, Mona Diab, Bonnie Dorr, Louise Guthrie, Magdalena Markowska, Smaranda Muresan, et al. 2022. Best: The belief and sentiment corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2460– 2467… find
  38. David Traum. 1994. A Computational Theory of Grounding in Natural Language Conversation. Ph.D. thesis, University of Rochester. find
  39. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. as a test set were calculated and presented in Table 12. To analyze the stability of the results, their average and standard deviation were determined in Table 13. find
  40. Annalisa Valle, Davide Massaro, Ilaria Castelli, and Antonella Marchetti. 2015. Theory of mind development in adolescence and early adulthood: The growing complexity of recursive thinking ability. Europe’s Journal of Psychology, 11(1):112–124. find
  41. Henry M. Wellman and David Liu. 2004. Scaling of theory-of-mind tasks. Child Development, 75(2):523– 541. find
  42. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. find
  43. Heinz Wimmer and Josef Perner. 1983. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 13(1):103–128. find
  44. Duzhen Zhang, Feilong Chen, and Xiuyi Chen. 2023. DualGATs: Dual graph attention networks for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7395–7408, Toronto, Canada. Association for Co… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/markowska-2023-finding-common-ground-annotating-predicting/