Elamur Research
☰
home / papers / 2021 / "I'll be there for you": The One with Understanding Indirect Answers

"I'll be there for you": The One with Understanding Indirect Answers

What it is about

Instead of crowd-writing questions and answers as Circa did, the authors mined 5,930 polar question–indirect answer pairs from the transcripts of a sitcom (228 episodes), removed bare yes/no-type words from the answers, sometimes added context to the question, and had three in-house annotators label them with Circa's relaxed classes (yes, no, conditional yes, neither). Agreement is high (Fleiss kappa 0.88; all three agree on 75%). The data is harder than Circa: longer, more varied answers, irony, mind changes within an answer. CNN classifiers reach 55% accuracy / 33 macro-F1 with GloVe and 64 / 49 with BERT features (majority 49 / 16); the answer alone again carries most of the signal. Training on each annotator's raw labels through a "crowd layer" raises macro-F1 (to 55 on dev), mostly on the rare conditional class, while adding Circa to training does not help.

Ideas

For Mova

languages/en/38-dialogue-and-discourse-markers.md §9 (questions and answers in dialogue), 31-speech-acts.md (agree/disagree). Code: prag/data/cues.tsv, prag/src/model.rs (weighted targets), prag/src/silver.rs (second teacher judgement). Measures: accuracy and macro-F1 on a tale question–reply set; rare-class macro-F1 in prag eval.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.codi-main.1.pdf

Summary

People easily understand replies to yes/no questions that never say "yes" or "no". Dialogue systems still struggle with these indirect answers. To support research on the problem, the authors built FRIENDS-QIA, an English corpus of 5,930 polar question–indirect answer pairs. Unlike the crowdsourced CIRCA corpus, the pairs were extracted by hand from transcripts of the sitcom "Friends". Explicit cue words were removed from answers, and missing context was added to short questions. Three of the authors then labeled every pair with CIRCA's six-class RELAXED scheme. Both the majority labels and each annotator's individual labels are released.

The experiments use CNN classifiers built on GloVe or BERT representations. They include cross-dataset tests with CIRCA and a deep learning from crowds (DLfC) layer that models each annotator's tendencies. BERT outperforms GloVe only when the model sees both the question and the answer. The crowd layer further improves macro F1 for the best models. The task remains hard, but the results suggest annotator disagreement carries useful information rather than mere noise.

Key points

  • FRIENDS-QIA draws on 228 episodes across 10 seasons of "Friends". Pairs whose answer was only a bare yes/no cue were discarded.
  • Inter-annotator agreement is high (Fleiss Kappa 0.8833). All three annotators agree on 75.02% of pairs, two agree on 23.39%, and all disagree on 1.59%.
  • The label set is CIRCA's RELAXED scheme: YES, NO, YES SUBJECT TO SOME CONDITIONS, NEITHER YES NOR NO, OTHER, N/A. The conditional-yes class is strongly underrepresented.
  • Scores on FRIENDS-QIA are lower than on CIRCA (34,268 pairs). The authors attribute this to the smaller size, open-domain sitcom dialogue with sarcasm and multi-sentence replies, and fewer annotations per item (three vs. five).
  • Model errors cluster on items where annotators disagreed. Category 3 pairs, which needed added context, are somewhat harder to classify.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.codi-main.1

@inproceedings{damgaard2021i,
  title = {{"I'll be there for you": The One with Understanding Indirect Answers}},
  author = {Damgaard, Cathrine and Toborek, Paulina and Eriksen, Trine and Plank, Barbara},
  booktitle = {Proceedings of the 2nd Workshop on Computational Approaches to Discourse},
  year = {2021},
  doi = {10.18653/v1/2021.codi-main.1},
  url = {https://doi.org/10.18653/v1/2021.codi-main.1}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find our card
  2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  4. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136–143, London, UK. Association for Computational Linguistics. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Nancy Green and Sandra Carberry. 1992. Conversational implicatures in indirect replies. In 30th Annual Meeting of the Association for Computational Linguistics, pages 64–71, Newark, Delaware, USA. Association for Computational Linguistics. find
  7. Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389–435. find
  8. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spe-jewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In Fifth European Conference on Speech Communication and Technology. find
  9. and Cathy Sotillo. 1993. The HCRC map task corpus: natural dialogue for speech recognition. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
  10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. find our card
  11. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. The Journal of Artificial Intelligence Research, Forthcoming. find
  12. Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. 2014. Learning from multiple annotators with varying expertise. Machine Learning, page 291–327. find
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. volume abs/1412.6980. ↗
  14. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “I’d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425, Online. Association for Computational Linguistics. find our card
  15. Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable annotations. In SAD/CrowdBias@ HCOMP, pages 19– 31. find
  16. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗ our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), page 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  18. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden. Association for Computational Linguistics. find
  19. Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2013. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428–1436. find
  20. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). Association for the Advancement of Artificial Intelligence. find
  21. Anna-Brita Stenström. 1984. Questions and responses in English conversation. Krieger Pub Co. Henry S. Thompson, Anne H. Anderson, Ellen Gurman Bard, Gwyneth Doherty-sneddon, Alison Newlands, find