Elamur Research⌕
☰
home / papers / 2023 / PaniniQA: Enhancing Patient Education Through Interactive Question Answering

PaniniQA: Enhancing Patient Education Through Interactive Question Answering

Paper · 2023 · Pengshan Cai, Zonghai Yao, Fei Liu et al.

Summary

Patients leaving hospital often struggle to grasp or remember their discharge instructions, and their misreadings usually go uncorrected. PaniniQA addresses this with a chatbot that quizzes patients about their own instructions. It follows the Prompt–Evaluate–Expand–Repeat (P.E.E.R.) cycle from Dialogic Reading.

The authors compare two ways of generating questions:

  • Direct: zero-shot prompting of GPT-3.5 (text-davinci-003) on the instruction text.
  • Enhanced: the same model, steered toward salient medical events and the correlational or causal links between them, such as a procedure and the condition it treats.

To support event extraction, they built a new expert-annotated dataset of events and relations over MIMIC-III discharge instructions. A ChatGPT-based module then judges each patient answer as correct, partially correct, or wrong, and replies with confirmation or clarification.

Automatic, GPT-4-based and human evaluations show that event-guided questions outperform directly generated ones. The feedback is mostly accurate, although the LLM still sometimes hallucinates. The work points to conversational quizzing as a practical complement to static patient portals.

Key points

  • Motivation: patients may immediately forget up to 80% of discharge information.
  • Models and resources: questions come from text-davinci-003, and answers are checked by gpt-3.5-turbo. The new event/relation annotations are built on MIMIC-III, and data and code are on GitHub (pengshancai/PaniniQA).
  • GPT-4 evaluation: generated questions scored around 4 or higher, and Q(Enhanced) beat Q(Direct). Answer feedback averaged 4.14 for Correctness and 4.01 for Education Potential.
  • Heuristic review: an MD student assessed 192 chatbot responses from 30 conversations. 86.4% were judged correct and 74.1% educationally useful.
  • Privacy: API calls were limited to synthetic discharge notes, each checked by physicians.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

Cite

Cite as Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang, Meghan Reilly, Huixue Zhou, Lingxi Li, Yi Cao, Alok Kapoor, Adarsha Bajracharya, Dan Berlowitz and Hong Yu. 2023. PaniniQA: Enhancing Patient Education Through Interactive Question Answering. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00616

@article{cai2023paniniqa,
  title = {{PaniniQA: Enhancing Patient Education Through Interactive Question Answering}},
  author = {Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang, Meghan Reilly, Huixue Zhou, Lingxi Li, Yi Cao, Alok Kapoor, Adarsha Bajracharya, Dan Berlowitz and Hong Yu},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00616},
  url = {https://doi.org/10.1162/tacl_a_00616}
}

References

64 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sabita Acharya, Barbara Di Eugenio, Andrew Boyd, Richard Cameron, Karen Dunn Lopez, Pamela Martyn-Nemeth, Carolyn Dickens, and Amer Ardati. 2018. Towards generating personalized hospitalization summaries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp… find
  2. Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noémie Elhadad. 2021. What’s in a summary? Laying the groundwork for advances in hospital-course summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  3. Griffin Adams, Han-Chin Shing, Qing Sun, Christopher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022. Learning to revise references for faithful summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4009–4027, Abu Dhabi, United Arab Emirates. Associati… find
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  5. Jordan Boyd-Graber and Benjamin Börschinger. 2020. What question answering can learn from trivia nerds. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422–7435, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .a… ↗
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  7. Pengshan Cai, Fei Liu, Adarsha Bajracharya, Joe Sills, Alok Kapoor, Weisong Liu, Dan Berlowitz, David Levy, Richeek Pradhan, and Hong Yu. 2022a. Generation of patient after-visit summaries to support physicians. In Proceedings of the 29th International Conference on Computational Linguistics, pages… find
  8. Pengshan Cai, Hui Wan, Fei Liu, Mo Yu, Hong Yu, and Sachindra Joshi. 2022b. Learning as conversation: Dialogue systems reinforced for information acquisition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  9. J. Harry Caufield, Yichao Zhou, Yunsheng Bai, David A. Liem, Anders O. Garlid, Kai-Wei Chang, Yizhou Sun, Peipei Ping, and Wei Wang. 2019. A comprehensive typing system for information extraction from clinical narratives. medRxiv, pages 19009118. https://doi.org/10.1101/19009118 — [doi:10.1101/19009118](https://doi.org/10.1101/19009118) ↗
  10. Yllias Chali and Sadid A. Hasan. 2015. Towards topic-to-question generation. Computational Linguistics, 41(1):1–20. https://doi.org/10.1162/COLI_a_00206 ↗
  11. Haw-Shiuan Chang, Zonghai Yao, Alolika Gon, Hong Yu, and Andrew McCallum. 2023. Revisiting the architectures like pointer networks to efficiently improve the next word distribution, summarization factuality, and beyond. arXiv preprint arXiv:2305.12289. https://doi.org/10.18653/v1/2023 .findings-acl.… ↗
  12. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Barbara Di Eugenio, Andrew Boyd, Camillo Lugaresi, Abhinaya Balasubramanian, Gail Keenan, Mike Burton, Tamara Goncalves Rezende Macieira, Jianrong Li, Yves Lussier, and Yves Lussier. 2014. PatientNarr: Towards generating patient-centric summaries of hospital stays. In Proceedings of the 8th Internat… find
  15. Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2067–2073, Copenhagen, Denmark. Association for Computational Linguistics. https://do… find
  16. Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866–874, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org /10.18653/v1/D17-… ↗
  17. Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, DaHyeon Choi, Chuning Yuan, and Chris Callison-Burch. 2022. A feasibility study of answer-agnostic question generation for education. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1919–19… find
  18. Maksim Eremeev, Ilya Valmianski, Xavier Amatriain, and Anitha Kannan. 2023. Injecting knowledge into language generation: A case study in auto-charting after-visit care instructions from medical dialogue. arXiv preprint arXiv:2306.03652. https://doi.org/10 .18653/v1/2023.acl-long.133 find
  19. Alexander Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. Template-based question generation from retrieved sentences for improved unsupervised question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4508–4513, On… find
  20. Alex Federman, Erin Sarzynski, Cindy Brach, Paul Francaviglia, Jessica Jacques, Lina Jandorf, Angela Sanchez Munoz, Michael Wolf, and Joseph Kannry. 2018. Challenges optimizing the after visit summary. International Journal of Medical Informatics, 120:14–19. https://doi.org/10.1016/j.ijmedinf .2018.… — [doi:10.1016/j.ijmedinf](https://doi.org/10.1016/j.ijmedinf) ↗
  21. Roberta Michnick Golinkoff, Erika Hoff, Meredith L. Rowe, Catherine S. Tamis-LeMonda, and Kathy Hirsh-Pasek. 2019. Language matters: Denying the existence of the 30-million-word gap has serious consequences. Child Development, 90(3):985–992. https://doi.org/10.1111/cdev.13128, PubMed: 30102419 — [doi:10.1111/cdev.13128](https://doi.org/10.1111/cdev.13128) ↗
  22. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pre-training for biomedical natural language processing. https://doi.org/10.1145/3458754 ↗
  23. Vince Hartman and Thomas R. Campion. 2022. A day-to-day approach for automating the hospital course section of the discharge summary. AMIA Annual Symposium Proceedings, 2022:216–225. find
  24. Michael Heilman and Noah A. Smith. 2010. Good question! Statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617, Los Angeles, California. Association for Comp… find
  25. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pub-MedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc…
  26. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016a. Mimic-iii, a freely accessible critical care database. Scientific Data, 3(1):1–9. https://doi.org/10.1038 /sdata.2016.35,…
  27. Roy P. C. Kessels. 2003. Patients’ memory for medical information. Journal of the Royal Society of Medicine, 96(5):219–222. https://doi.org/10.1177/014107680309600504, PubMed: 12724430 — [doi:10.1177/014107680309600504](https://doi.org/10.1177/014107680309600504) ↗
  28. Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2018. Improving neural question generation using answer separation. CoRR, abs/1809.02393. find
  29. Sunjae Kwon, Zonghai Yao, Harmon S. Jordan, David A. Levy, Brian Corner, and Hong Yu. 2022. Medjex: A medical jargon extraction model with wiki’s hyperlink span and contextualized masked language model score. arXiv preprint arXiv:2210.05875. https://doi.org/10.18653/v1/2022.emnlp-main .805, PubMed:… ↗
  30. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. https://doi.org/10 .1093/bioinformatics/btz682, PubMed: 31501885
  31. Eric Lehman, Vladislav Lialin, Katelyn Edelwina Legaspi, Anne Janelle Sy, Patricia Therese Pile, Nicole Rose Alberto, Richard Raymund Ragasa, Corinna Victoria Puyat, Marianne Katharina Taliño, Isabelle Rose Alberto, Pia Gabrielle Alfonso, Dana Moukheiber, Byron Wallace, Anna Rumshisky, Jennifer Lia… find
  32. Rosemary Lever and Monique Sénéchal. 2011. Discussing stories: On how a dialogic reading intervention improves kindergartners’ oral narrative construction. Journal of Experimental Child Psychology, 108(1):1–24. https://doi.org/10.1016/j.jecp.2010.07.002, PubMed: 20828708 — [doi:10.1016/j.jecp.2010.07.002](https://doi.org/10.1016/j.jecp.2010.07.002) ↗
  33. Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146–157. https://doi.org/10.18653/v1/2020 .c… ↗
  34. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
  35. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. https://doi.org/10 .48550/arXiv.2301.13688
  36. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. https://d…
  37. Suzanne E. Mol, Adriana G. Bus, Maria T. De Jong, and Daisy J. H. Smeets. 2008. Added value of dialogic parent–child book readings: A meta-analysis. Early Education and Development, 19(1):7–26. https://doi.org/10 .1080/10409280701838603 find
  38. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  39. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  40. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
  41. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrQA: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368, Brussels, Belgium. Association for Computatio…
  42. Stacey Pylman and Amy Ward. 2020. 12 tips for effective questioning in medical education. Medical Teacher, 42(12):1330–1336. https://doi.org/10.1080/0142159X .2020.1749583, PubMed: 32297833 — [doi:10.1080/0142159x](https://doi.org/10.1080/0142159x) ↗
  43. Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In LREC. Citeseer. find
  44. Preethi Raghavan, Jennifer J. Liang, Diwakar Mahajan, Rachita Chandra, and Peter Szolovits. 2021. emrKBQA: A clinical knowledge-base question answering dataset. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 64–73, Online. Association for Computational Linguistics. http… find
  45. Bhanu Pratap Singh Rawat, Wei-Hung Weng, So Yeon Min, Preethi Raghavan, and Peter Szolovits. 2020. Entity-enriched neural models for clinical question answering. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 112–122, Online. Association for Computational Ling… find
  46. Claude Richard, Emma Glaser, and Marie-Thérèse Lussier. 2017. Communication and patient participation influencing patient recall of treatment discussions. Health Expectations, 20(4):760–770. https://doi.org/10.1111/hex.12515, PubMed: 27868327 ↗
  47. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  48. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615–4629, Online. Association for Computational L… find
  49. Sarvesh Soni and Kirk Roberts. 2020. Evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5532–5538, Marseille, France. European Language Resourc… find
  50. Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. 2020. On the importance of diversity in question generation for QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5651–5656, Online. Association for Computational… find
  51. Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653/v1/2021.emnlp-main.388 ↗
  52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201 .11903. https://doi.org/10.48550/arXiv.2201.11903 ↗
  53. Grover J. Whitehurst. 2002. Dialogic reading: An effective way to read aloud with young children. https://www.readingrockets.org/article/dialogic-reading-effective-way -read-aloud-young-children. ↗
  54. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA – an authenti…
  55. Junda Wang, Zonghai Yao, Avijit Mitra, Samuel Osebe, Zhichao Yang, and Hong Yu. 2023a. UMASS BioNLP at MEDIQA-chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations? In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 460–471, Toront… find
  56. Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Li, Mo Yu, and Ying Xu. 2022a. It is AI’s turn to ask humans a question: Question-answer pair generation for children’s story books. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  57. Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023b. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. arXiv preprint arXiv:2305.13412. https://doi.org/10.18653/v1/2023.acl-long.482 ↗
  58. Zonghai Yao, Yi Cao, Zhichao Yang, Vijeta Deshpande, and Hong Yu. 2022b. Extracting biomedical factual knowledge using pretrained language model and electronic health record context. arXiv preprint arXiv:2209.07859. — [arXiv:2209.07859](https://arxiv.org/abs/2209.07859) find
  59. Himali Weerahandi, Boback Ziaeian, Robert L. Fogerty, Grace Y. Jenq, and Leora I. Horwitz. 2018. Predictors for patients understanding reason for hospitalization. PLoS One, 13(4):e0196479. https://doi.org/10 .1371/journal.pone.0196479, PubMed: 29702676 find
  60. Zonghai Yao, Yi Cao, Zhichao Yang, and Hong Yu. 2022c. Context variance evaluation of pretrained language models for prompt-based biomedical knowledge probing. arXiv preprint arXiv:2211.10265. — [arXiv:2211.10265](https://arxiv.org/abs/2211.10265) find
  61. Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun. 2020a. Clinical reading comprehension: A thorough analysis of the emrQA dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4474–4486, Online. Association for Computational Linguistics. https://… find
  62. Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun. 2020b. Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering. arXiv preprint arXiv:2010.16021. https://doi.org/10.1109/BIBM52615.2021.9669300 — [doi:10.1109/bibm52615.2021.9669300](https://doi.org/10.1109/bibm52615.2021.9669300) ↗
  63. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu…
  64. Jane Y. Zhao, Buer Song, Edwin Anand, Diane Schwartz, Mandip Panesar, Gretchen P. Jackson, and Peter L. Elkin. 2017. Barriers, facilitators, and solutions to optimal patient portal and personal health record use: A systematic review of the literature. In AMIA Annual Symposium Proceedings, volume 201… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cai-2023-paniniqa-enhancing-patient-education-through/