Elamur Research
☰
home / papers / 2021 / Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

What it is about

The paper studies unanswerable questions in Natural Questions. In a hand analysis of 100 of them, 30% are unanswerable because a presupposition of the question cannot be verified in the linked document («What is the stock symbol for Mars candy?» presupposes that there is one). Most such presuppositions come from the wh-word (19 of 30) and the definite article (10 of 30). In a user study, answers that deny the failed presupposition («…because we could not verify that any linguist invented the lightbulb») were preferred over a bare «unanswerable», a random related sentence, and an open-domain system's answer. The authors split the task into three steps. A rule-based generator over parse trees turns triggers into presuppositions: wh-word → «some X did Y», definite → existence and uniqueness, possessive → «A has B», factive verb → complement, temporal clause → event, counterfactual. It is 83% valid; errors come from compounds like «bachelor's degree» and from embedding under believe. Verification against the document uses NLI or fact-checking models, with only modest gains (macro-F1 0.60 vs 0.44 for majority). Template explanations follow.

Ideas

For Mova

languages/en/33-implicature-and-presupposition.md §6–7 (triggers, projection, accommodation), languages/en/26-questions.md; world/src/knowledge.rs and world knowledge diagnose (docs/mmm-diagnosis.md), prag/src/infer.rs, world/src/talk.rs; measures: the share of test questions flagged "presupposition unsupported" per course, and whether excluding them changes accuracy on the rest (it should not).

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.304.pdf

Summary

Many QA benchmarks contain questions with no answer. Systems usually just label these "unanswerable" or, worse, give an answer that accepts a false premise, such as naming Edison for a lightbulb-inventing linguist. The authors argue that many such questions fail because of an unverifiable presupposition, an implicit background assumption. They show this with a manual analysis of Natural Questions (NQ). A user preference study found that people prefer oracle answers that point out the failed presupposition over the outputs of existing closed-book QA systems.

They then split the task into three stages:

  • Presupposition generation, based on triggers such as wh-words, the definite article, factives and possessives.
  • Verification against the source document, treated as an NLI-style or fact-checking problem.
  • Explanation generation.

Verification turned out to be the hardest stage. A hybrid of a strong NLI model, extra fine-tuning and rules improved on the baseline but did not solve it. Adding presuppositions and their verification labels to the input of the ETC QA model gave small gains. This suggests that presupposition-aware QA can produce more cooperative and informative responses.

Key points

  • In 100 sampled NQ dev wh-questions where all five annotations were Null, 30% were attributed to unverifiable presuppositions when both annotators agreed (Cohen's κ = 0.586). This projects to about 21% of unanswerable NQ questions overall, and a further 21% could be partially explained this way.
  • The failed presuppositions came mostly from question words (19/30) and the definite article "the" (10/30), with one from a factive verb.
  • The user study compared four answer types:
    • a plain "unanswerable" statement;
    • a presupposition-failure explanation;
    • an extractive explanation;
    • a DPR-based (Karpukhin et al., 2020) rewrite.
  • With ETC, augmentation gave modest improvements in Average F1 and Unanswerability Classification. Under McNemar's test, the improvements over the baseline held at roughly 86% (flat input) and 89% (structured input) confidence.
  • Prediction changes that led to correct answers were asymmetric. Switches from answerable to unanswerable made up 11.9% of unanswerable questions, while switches the other way made up 6.7% of answerable ones. Noisy verification labels remain the main bottleneck.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.304 · arXiv 2101.00391

@inproceedings{kim2021which,
  title = {{Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering}},
  author = {Kim, Najoung and Pavlick, Ellie and Ayan, Burcu Karagol and Ramachandran, Deepak},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.304},
  eprint = {2101.00391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.304}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Márta Abrusán. 2011. Presuppositional and negative islands: a semantic account. Natural Language Semantics, 19(3):257–321. find
  2. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268–284, Online. Association for Computational Linguistics. find
  3. Akari Asai and Eunsol Choi. 2020. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval. arXiv:2010.11915. ↗
  4. Seohyun Back, Sai Chetan Chinthakindi, Akhil Kedia, Haejun Lee, and Jaegul Choo. 2020. NeurQuRI: Neural question requirement inspector for answerability prediction in machine reading comprehension. In International Conference on Learning Representations. find
  5. David Beaver. 1997. Presupposition. In Handbook of Logic and Language, pages 939–1008. Elsevier. find
  6. Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. 2018. Let’s do it “again”: A first computational approach to detecting adverbial presupposition triggers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2747– 2755, Melbourne, Australia. Association for Computational Linguistics. find our card
  7. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 845–855, Melbourne, Australia. Association for Computational Linguistics. find our card
  8. David Clausen and Christopher D. Manning. 2009. Presupposed content and entailments in natural language inference. In Proceedings of the 2009 Workshop on Applied Textual Inference (TextInfer), pages 70–73, Suntec, Singapore. Association for Computational Linguistics. find
  9. Elizabeth Coppock and David Beaver. 2012. Weak uniqueness: The only difference between definites and indefinites. In Semantics and Linguistic Theory, volume 22, pages 527–544. find
  10. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. find our card
  11. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. arXiv:2101.02235. ↗
  12. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, page 25–30, New York, NY, USA. Association for Computing Machinery. find
  13. Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. 2019. Read+ verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6529– 6537. find
  14. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Online. Association for Computational Linguistics. ↗ our card
  16. Lauri Karttunen. 2016. Presupposition: What went wrong? In Semantics and Linguistic Theory, volume 26, pages 705–731. find
  17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. find our card
  18. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2020. QED: A framework and dataset for explanations in question answering. arXiv:2009.06354. ↗
  19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations. find
  20. Stephen C. Levinson. 1983. Pragmatics, Cambridge Textbooks in Linguistics, pages 181–184, 194. Cambridge University Press. find
  21. Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342–7351, Online. Association for Computational Linguistics. find
  22. Galina Tremper and Anette Frank. 2011. Extending fine-grained semantic relation classification to presupposition relations between verbs. Bochumer Linguistische Arbeitsberichte. find
  23. Randall Munroe. 2020. Learning new things from Google. https://twitter.com/xkcd/status/ 1333529967079120896. Accessed: 2021-02-01. ↗
  24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. find our card
  25. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia. Association for Computational Linguistics. find our card
  26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  27. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Linguistics. ↗ our card
  28. Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318–2323, Minneapolis, Minnesota. Association for Computational Linguistics. find
  29. Rob A. Van der Sandt. 1992. Presupposition projection as anaphora resolution. Journal of Semantics, 9(4):333–377. find
  30. Philippe Schlenker. 2008. Presupposition projection: Explanatory strategies. Theoretical Linguistics, 34(3):287–316. find
  31. Bernhard Schwarz and Alexandra Simonenko. 2018. Decomposing universal projection in questions. In Sinn und Bedeutung 22, volume 22, pages 361–374. find
  32. Florian Schwarz. 2016. Experimental work in presupposition and presupposition projection. Annual Review of Linguistics, 2(1):273–292. find
  33. Mandy Simons. 2013. Presupposing. Pragmatics of Speech Actions, pages 143–172. find
  34. Peter F. Strawson. 1950. 59(235):320–344. On referring. find
  35. Mind, Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2020. Retrospective reader for machine reading comprehension. arXiv:2001.09694. ↗
  36. Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to ask unanswerable questions for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4238–4248, Florence, Italy. Association for Computational Linguistics. find