Elamur Research
☰
home / papers / 2023 / CREPE: Open-Domain Question Answering with False Presuppositions

CREPE: Open-Domain Question Answering with False Presuppositions

What it is about

CREPE has 8,400 «explain like I'm five» questions from Reddit. Annotators, guided by the most upvoted answer, mark whether the question rests on a false presupposition and, if so, write the presupposition and a correction. About 25% of real information-seeking questions contain one. A sample of 50 shows the kinds: a false predicate (30%, «how is current stored in power plants?»), false properties (22%), false causal links between facts (22%), false clauses (14%, «if water has to be 100° to become steam…»), false existentials (6%) and over-generalisations (4%). The corrective answers usually start with a lexical signal: negation in 70% of signalled cases, or actually, just, really, «you are mistaken». Baselines with retrieval and pretrained models detect the explicit cases moderately well but fail on implicit ones. The main bottleneck is retrieving evidence that contradicts the presupposition, not reading it.

Ideas

For Mova

languages/en/33-implicature-and-presupposition.md §6–7, languages/en/26-questions.md; world/src/knowledge.rs, world knowledge diagnose, prag/src/feats.rs; measures: count of flagged questions per course, disagree F1 in prag eval.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-long.583.pdf

Summary

People who ask about unfamiliar subjects often build mistaken assumptions into their questions. Most QA benchmarks ignore this because they assume every question has a valid answer or is unanswerable only for lack of evidence. CREPE addresses this gap with 8,400 real questions from the ELI5 subreddit. Annotators marked whether each question contains a false presupposition and, if so, wrote the presupposition and its correction. To keep costs low and quality high, they relied on the most upvoted community comment as evidence rather than searching the web themselves. About a quarter of the questions contain such mistaken assumptions.

The authors define two subtasks: detecting a false presupposition and writing it with a correction. These are run in two settings, one with the question alone and one that also supplies the gold comment. Baselines combining passage retrieval with pre-trained language models catch explicit errors reasonably well. They falter on implicit assumptions and on explaining why an assumption is wrong, largely because retrieving the right evidence is hard. The benchmark pushes QA toward handling realistic, imperfect user questions.

Key points

  • Data: 8,400 Reddit ELI5 questions, 25% of which contain false presuppositions, are split by posting date:
    • train: 2011–2018
    • dev: January–June 2019
    • test: July–December 2019
  • Tasks and tracks:
    • Two subtasks: detection and writing.
    • Two tracks: the main track (question only) and the GOLD-COMMENT track (question plus comment).
  • Baselines: three approaches built on pre-trained LMs: one that sees only the question, one that relies on nearest neighbors, and one that uses retrieval (Krishna et al., 2021).
  • Evaluation:
    • SentBERT scores look high but are misleading.
    • Human ratings on 200 test items (0–3 scale for fluency, presupposition, correction and consistency) show that about half of the generated "false" presuppositions are actually true.
  • Further findings:
    • A small case study with GPT-3 text-davinci-002 suggests the model rarely flags or corrects false presuppositions.
    • A similar 25% rate of false presuppositions appears in questions about NLP papers.
    • Code and data: github.com/velocityCavalry/CREPE.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.583 · arXiv 2211.17257

@inproceedings{yu2023crepe,
  title = {{CREPE: Open-Domain Question Answering with False Presuppositions}},
  author = {Yu, Xinyan and Min, Sewon and Zettlemoyer, Luke and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.583},
  eprint = {2211.17257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.583}
}

References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai and Eunsol Choi. 2021. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the Association for Computational Linguistics. find
  2. FairScale authors. 2021. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https://github.com/f acebookresearch/fairscale. ↗
  3. David I. Beaver, Bart Geurts, and Kristie Denlinger. 2014. Presupposition. The Stanford Encyclopedia of Philosophy. find
  4. Rahul Bhagat and Eduard Hovy. 2013. Squibs: What is a paraphrase? Computational Linguistics. ↗ our card
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems. find
  6. Isabel Cachola, Eric Holgate, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2018. Expressively vulgar: The socio-dynamics of vulgarity and its effects on sentiment analysis in social media. In Proceedings of International Conference on Computational Linguistics. In Proceedings of Empirical Methods in Natural Language Processing. find
  7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  8. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  9. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In Proceedings of the International Conference on Learning Representations. find
  10. Marie Duží and Martina Číhalová. 2015. Questions, answers, and presuppositions. Computación y Sistemas. find
  11. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics. ↗ our card
  12. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  13. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the European Chapter of the Association for Computational Linguistics. ↗ our card
  14. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the Association for Computational Linguistics. ↗ our card
  15. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. find our card
  16. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint. find
  17. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
  18. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. find our card
  19. S. Jerrold Kaplan. 1978. Indirect responses to loaded questions. In Theoretical Issues in Natural Language Processing-2. ↗
  20. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the Association for Computational Linguistics. find our card
  21. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ↗ our card
  22. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics. find our card
  23. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics. ↗ our card
  24. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the European Chapter of the Association for Computational Linguistics. biguous open-domain questions. In Proceedings of Empirical Methods in Natural Language Processing. ↗
  25. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint. find our card
  26. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the Association for Computational Linguistics. ↗ our card
  27. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ↗ our card
  28. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation. ↗ our card
  29. Christopher Potts. 2015. Presupposition and implicature. The handbook of contemporary semantic theory. find
  30. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus-sière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. find our card
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. find
  32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint. find
  33. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of Empirical Methods in Natural Language Processing. ↗
  34. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In Proceedings of the International Conference on Learning Representations. find
  35. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am-Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the Association for Computational Linguistics. find our card
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  37. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of Empirical Methods in Natural Language Processing. ↗
  38. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the Association for Computational Linguistics. ↗ our card
  39. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Uribe, Liam Fedus, Luke Metz, et al. 2022. ChatGPT: Optimizing language models for dialogue. find
  40. Robert Stalnaker, Milton K Munitz, and Peter Unger. 1977. Pragmatic presuppositions. In Proceedings of the Texas conference on per˜ formatives, presuppositions, and implicatures. find
  41. Peter F Strawson. 1950. On referring. Mind. find
  42. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Language Resources and Evaluation Conference. ↗
  43. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. ↗
  44. Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022. How do we answer complex questions: Discourse structure of long-form answers. In Proceedings of the Association for Computational Linguistics. ↗
  45. Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371– 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find