Elamur Research⌕
☰
home / papers / 2024 / Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering

Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering

Summary

People asking health questions often build in unstated beliefs, and some of those beliefs are wrong or risky. In maternal and infant care, a QA system that only answers the literal question can leave such beliefs intact. The authors frame these hidden beliefs as pragmatic inferences, meaning presuppositions and implicatures.

To study them, public health experts annotated 500 questions from three sources: the authors' deployed maternal-health system ROSIE, parenting subreddits, and Natural Questions. For each question, the experts wrote down the assumptions and implications it carried. They then judged how plausible each one was and checked whether it was true against trusted web documents.

The analysis shows that false beliefs appear more often as implicatures than as presuppositions. The authors then gave the expert-written inferences to an existing retrieval-plus-reader QA pipeline. Experts found the resulting answers more helpful when a question contained one or more false inferences that had been judged highly plausible.

The work matters because it shows that "questioning the question" can reduce the spread of harmful health misconceptions. It also shows that current LLMs struggle to produce such inferences on their own.

Key points

  • The dataset holds 2,727 expert-annotated pragmatic inferences over 500 evaluation questions, plus 150 development questions:
    • ROSIE: 200 questions
    • Reddit: 200 questions
    • Natural Questions: 100 questions
  • Reddit questions had the largest share of false or subjective inferences, at about 31%. ROSIE and NQ were near 20–22%.
  • ROSIE retrieves from a corpus of roughly 408,000 trusted web documents, split into 100-token passages.
  • Answers informed by inferences beat default answers only on questions with a highly plausible (plausibility = 5) false inference. Across all questions, the two systems were rated about the same.
  • The authors tested whether GPT-3.5 with in-context learning could generate the inferences, with author agreement κ = 0.88 on matching. GPT-3.5 missed 63% of the human-written inferences, including 53% of presuppositions and 71% of implicatures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Implicature

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-long.403 · arXiv 2311.09542

@inproceedings{srikanth2024pregnant,
  title = {{Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering}},
  author = {Srikanth, Neha and Sarkar, Rupak and Mane, Heran and Aparicio, Elizabeth and Nguyen, Quynh and Rudinger, Rachel and Boyd-Graber, Jordan},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.403},
  eprint = {2311.09542},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.403}
}

References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dorit Abusch. 2002. Lexical alternatives as a source of pragmatic presuppositions. In Semantics and linguistic theory, volume 12, pages 1–19. find
  2. Nicholas Allott. 2018. Conversational implicature. find
  3. Akari Asai and Eunsol Choi. 2021. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  4. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260. find
  5. Adrien Barbaresi. 2021. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… find
  6. David Ian Beaver. 1997. Presupposition. In Handbook of logic and language, pages 939–1008. Elsevier. find
  7. Gay Becker and Edwina Newsom. 2003. Socioeconomic status and dissatisfaction with health care among chronically ill african americans. American journal of public health, 93(5):742–748. find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… find our card
  10. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  11. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for C… find our card
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In Proceedings of the 29th International Conference on Computational Linguistics, pages 779–785, Gyeongju, Republic of Korea. International Committee on Computat… find our card
  14. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… find our card
  15. Najoung Kim, Phu Mon Htut, Sam Bowman, and Jackson Owen Petty. 2022. (qa)2: Question answering with questionable assumptions. ArXiv, abs/2212.10003. find
  16. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer… find our card
  17. Todor Koev. 2018. Notions of at-issueness. Language and Linguistics Compass, 12(12):e12306. find
  18. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  19. Stephen C Levinson, Stephen C Levinson, and S Levinson. 1983. Pragmatics. Cambridge university press. find
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  21. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114… find our card
  22. Heran Y Mane, Amara Channell Doig, Francia Ximena Marin Gutierrez, Michelle Jasczynski, Xiaohe Yue, Neha Pundlik Srikanth, Sourabh Mane, Abby Sun, Rachel Ann Moats, Pragat Patel, et al. 2023. Practical guidance for the development of rosie, a health education question-and-answer chatbot for new moth… find
  23. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In ACL. find
  24. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  25. Adam Palanica, Peter Flaschner, Anirudh Thomman-dram, Michael Li, and Yan Fossat. 2019. Physicians’ perceptions of chatbots in health care: cross-sectional web-based survey. Journal of medical Internet research, 21(4):e12887. find
  26. Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 349–366, Online.… find our card
  27. Stanley Peters. 2016. Speaker commitments: presupposition. In Semantics and Linguistic Theory, pages 1083–1098. find
  28. Christopher Potts. 2004. The logic of conventional implicatures, volume 7. OUP Oxford. find
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  30. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. find our card
  31. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  32. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. find our card
  33. Ben Shneiderman. 1992. Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions on graphics (TOG), 11(1):92–99. find
  34. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2022. Prompting gpt-3 to be reliable. find our card
  35. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. CoRR, abs/2004.09297. find our card
  36. Robert Stalnaker, Milton K Munitz, and Peter Unger. 1977. Pragmatic presuppositions. In Proceedings of the Texas conference on per˜ formatives, presuppositions, and implicatures. Arlington, VA: Center for Applied Linguistics, pages 135–148. ERIC. find
  37. Robert S Taylor. 1962. The process of asking questions. American documentation, 13(4):391–396. find
  38. Gautier TIzacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. In Transactions on Machine Learning Research. find our card
  39. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada. Association for C… find
  40. Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Crepe: Open-domain question answering with false presuppositions. ArXiv, abs/2211.17257. find our card
  41. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A grammar-based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074–2085, Online. Association for Computational Ling… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/srikanth-2024-pregnant-questions-importance-pragmatic-awareness/