Elamur Research
☰
home / papers / 2020 / "I'd rather just go to bed": Understanding Indirect Answers

"I'd rather just go to bed": Understanding Indirect Answers

What it is about

The paper builds Circa, 34,268 pairs of a polar question and an indirect answer (no yes/no words), in ten everyday scenarios, written and labelled by crowd workers. Each pair has five judgements of how the asker would read it: yes, probably yes, yes under a condition, no, probably no, in the middle, unsure, other (relaxed: yes 48.5%, no 37.5%, conditional yes 7.5%, middle 2.8%; Fleiss kappa 0.61 strict, 0.76 relaxed). A mined sample of movie dialogues shows only 53% of answers to yes/no questions are direct. BERT models reach 88% (relaxed, 4 classes) and 85% (strict, 6) when the scenario was seen in training and 6–10 points less on an unseen one; the answer alone already gives 82%, the question alone 56%. An off-the-shelf NLI model, used by turning the question into a declarative and reading entailment as yes and contradiction as no, is good on no and poor on yes.

Ideas

For Mova

languages/en/38-dialogue-and-discourse-markers.md §9 (questions and answers in dialogue), 33-implicature-and-presupposition.md §5 (Relevance inferences), 31-speech-acts.md (agree/disagree). Code: prag/src/frames.rs (statement_order, frame matching), world/src/talk.rs, prag/data/cues.tsv. Measures: accuracy on a tale dev set of question–reply pairs (relaxed four classes), agree/disagree F1 in prag eval. Circa itself only as a test set, and only if its licence allows.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.601.pdf

Summary

People often answer yes/no questions without saying "yes" or "no". For example, someone might reply "I'm starving" when asked if they are hungry. Dialog systems need to work out the intended meaning of such replies, but earlier studies of this were small and mostly not data-driven. The authors built Circa, the first large English corpus of polar questions paired with indirect answers. They collected it through a multi-step crowdsourcing process grounded in 10 hand-designed everyday social scenarios. Workers wrote the questions, wrote answers that avoided explicit yes/no words, and labeled what each answer meant. The labels go beyond yes and no to include conditional, uncertain and middle-ground replies. The authors then fine-tuned BERT classifiers on question–answer pairs, testing transfer from entailment (MNLI), BoolQ and discourse data. The best model reaches about 88% accuracy on a 4-way label set and about 85% on a 6-way set. Accuracy falls noticeably when a scenario is absent from training. The resource therefore sets a benchmark for pragmatic inference in conversation and shows that models still struggle with it.

Key points

  • Circa corpus: 34,268 question–answer pairs built from 3,431 unique questions, each with up to 10 answers, across 10 dialog scenarios. Examples include food preferences and a friend switching jobs.
  • Label sets:
    • A relaxed set of 4 classes: yes, no, conditional yes, in the middle.
    • A strict set of 6 classes, which adds probably yes and probably no.
  • Models: BERT-YN, BERT-MNLI-YN, BERT-DIS-YN and BERT-BOOLQ-YN.
    • BERT-MNLI-YN performs best, with 88.2% test accuracy on relaxed labels and about 85% on strict labels in the matched setting.
    • Its gain over BERT-YN is statistically significant only for the strict labels.
  • Answer text vs. joint input: The answer text alone gives roughly 78–82% accuracy, but modeling the question and answer together is significantly better. In leave-one-scenario-out tests, accuracy is 6–10 points lower, dropping to 74% at worst for strict labels.
  • Motivation and hardest cases:
    • Among 6,327 yes/no question pairs mined from the Cornell Movie Dialog Corpus, only 53% had direct answers.
    • "In the middle" is the hardest class to predict.
    • About 69% of "probably no" items are misclassified as "no".

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.601 · arXiv 2010.03450

@inproceedings{louis2020i,
  title = {{"I'd rather just go to bed": Understanding Indirect Answers}},
  author = {Louis, Annie and Roth, Dan and Radlinski, Filip},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.601},
  eprint = {2010.03450},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.601}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a Human-Like Open-Domain Chatbot. arXiv preprint arXiv:2001.09977. ↗ our card
  2. Penelope Brown and Stephen C Levinson. 1978. Universals in Language Usage: Politeness Phenomena. In E.N. Goody, editor, Questions and Politeness: Strategies in Social Interaction, pages 56–311. Cambridge University Press. find
  3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question Answering in Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. find our card
  4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. find our card
  5. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  7. Nancy Green and Sandra Carberry. 1999. Interpreting and Generating Indirect Answers. Computational Linguistics, 25(3):389–435. find
  8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. find
  9. Julia B. Hirschberg. 1985. A Theory of Scalar Implicature. Ph.D. thesis, University of Pennsylvania. find
  10. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you Predict Responses to Yes/No Questions? Yes, No, and Stuff. In Fifth European Conference on Speech Communication and Technology. find
  11. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a Simple Yes or No: Uncertainty in Indirect Answers. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 136–143. find
  12. Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2010. Was it Good? It was Provocative. Learning the Meaning of Scalar Adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176. find our card
  13. Allen Nie, Erin Bennett, and Noah D. Goodman. 2019. DisSent: Learning Sentence Representations from Explicit Discourse relations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Volume 1: Long Papers, pages 4497–4510. find
  14. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis-Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. find
  15. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A Conversational Question Answering Challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find our card
  16. Deborah Rossen-Knill, Beverly Spejewski, Beth Ann Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/No Questions and Answers in the Map Task Corpus. Technical report, University of Pennsylvania. find
  17. Anna-Brita Stenström. 1984. Questions and Responses in English Conversation: Lund Studies in English. Claes Schaar and Jan Svartvik, editors, Lund Studies in English 68, CWK Gleerup, Malmo Sweden. find
  18. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. the Proceedings of ICLR. find our card
  19. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. find
  20. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation. arXiv preprint arXiv:1911.00536. ↗ our card