Elamur Research⌕
☰
home / papers / 2018 / Task-oriented Dialogue System for Automatic Diagnosis

Task-oriented Dialogue System for Automatic Diagnosis

Paper · 2018 · Zhongyu Wei, Qianlong Liu, Baolin Peng et al.

Summary

The paper tackles automatic disease diagnosis when a patient's initial description is incomplete. Diagnostic models built on electronic health records are expensive to train and hard to transfer across diseases. The authors therefore propose a task-oriented dialogue system that asks patients follow-up questions to gather more symptoms. They built a Chinese dataset from the pediatric section of an online healthcare platform (Baidu Muzhi). It covers four childhood diseases, and each record pairs a patient's self-report (explicit symptoms) with the doctor–patient conversation (implicit symptoms). Annotators marked symptom spans using BIO tags and recorded whether each symptom was present. Medical experts then mapped the spans to SNOMED CT concepts. The system focuses on the dialogue manager: a user simulator samples user goals from the data, and a DQN policy trained with reinforcement learning decides which symptom to ask about next or which disease to name. Adding implicit symptoms clearly improves classifier accuracy, and the DQN agent beats random and rule-based agents. This shows that conversational symptom collection is a practical route to automatic diagnosis.

Key points

  • Dataset: 710 user goals across four diseases: infantile diarrhea (200), children functional dyspepsia (150), upper respiratory infection (160) and children's bronchitis (200).

    • Annotators identified 144 unique normalized symptoms; only the 67 occurring at least 10 times were kept to shrink the action space.
    • Inter-annotator Cohen's kappa was 71% for self-reports and 67% for conversations.
    • The data split is 568 user goals for training and 142 for testing.

  • Value of implicit symptoms: an SVM using both explicit and implicit symptoms (SVM-ex&im) reached 0.71 overall accuracy, compared with 0.59 for explicit symptoms only (SVM-ex).

  • Dialogue agents on 5K simulated dialogues:

    AgentSuccess rateAverage rewardAverage turns
    Random0.06−24.3617.51
    Rule-based0.23−13.7817.00
    DQN0.6520.515.11

    The DQN agent outperforms SVM-ex but still trails SVM-ex&im.

  • Setup:

    • Dialogues are capped at 22 turns.
    • Rewards are +44 for a correct diagnosis and −22 for a failure, plus a −1 penalty per turn.
    • Natural language understanding and generation use templates.
    • The replay buffer is warm-started with experiences from the rule-based agent.

  • Authorship: the first author is Qianlong Liu; Zhongyu Wei is the corresponding author. The paper appeared as an ACL 2018 short paper (pp. 201–207), and the authors released the dataset publicly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China65 works · 19,402 citations🏢BaiduBeijing · China40 works · 14,627 citations🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksHealth Professions · 14 worksHealth Information Management · 9 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 401 works

Cite

Cite as Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong and Xiangying Dai. 2018. Task-oriented Dialogue System for Automatic Diagnosis. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2033

@inproceedings{wei2018task,
  title = {{Task-oriented Dialogue System for Automatic Diagnosis}},
  author = {Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong and Xiangying Dai},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2033},
  url = {https://doi.org/10.18653/v1/p18-2033}
}

References

22 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017b. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2… find
  2. Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. 2016. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association 24(2):361–370. find
  3. Heriberto Cuayahuitl, Simon Keizer, Oliver Lemon, et al. 2015. Strategic dialogue management via deep reinforcement learning. CoRR . find
  4. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volum…
  5. Finale Doshi-Velez, Yaorong Ge, and Isaac Kohane. 2014. Comorbidity clusters in autism spectrum disorders: an electronic health record time-series analysis. Pediatrics 133(1):e54–e63. find
  6. Siddhartha R Jonnalagadda, Abhishek K Adupa, Ravi P Garg, Jessica Corona-Cox, and Sanjiv J Shah. 2017. Text mining of the electronic health record: An information extraction approach for automated identification and subphenotyping of hfpef patients for clinical trials. Journal of cardiovascular tran… find
  7. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review 21(2):97–126. find
  8. Chaitanya Shivade, Preethi Raghavan, Eric Fosler-Lussier, Peter J Embi, Noemie Elhadad, Stephen B Johnson, and Albert M Lai. 2013. A review of approaches to identifying patient phenotype cohorts using electronic health records. Journal of the American Medical Informatics Association 21(2):221–230. find
  9. Hao-Cheng Kao, Kai-Fu Tang, and Edward Y Chang. 2018. Context-aware symptom checking for disease diagnosis using hierarchical reinforcement learning . find
  10. Li Li, Wei-Yi Cheng, Benjamin S Glicksberg, Omri Gottesman, Ronald Tamler, Rong Chen, Erwin P Bottinger, and Joel T Dudley. 2015. Identification of type 2 diabetes subgroups through topological analysis of patient similarity. Science translational medicine 7(311):311ra174–311ra174. find
  11. Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management https://arxiv.org/abs/1606.02689. ↗
  12. Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). volume 1, pages 733–743.
  13. Kai-Fu Tang, Hao-Cheng Kao, Chun-Nan Chou, and Edward Y Chang. 2016. Inquire and diagnose: Neural symptom checking ensemble using deep reinforcement learning. In Proceedings of NIPS Workshop on Deep Reinforcement Learning. find
  14. Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2017. End-to-end optimization of task-oriented dialogue model with deep reinforcement learning https://arxiv.org/abs/1711.10712. — [arXiv:1711.10712](https://arxiv.org/abs/1711.10712) ↗
  15. Huaixiao Tou, Lu Yao, Zhongyu Wei, Xiahai Zhuang, and Bo Zhang. 2018. Automatic infection detection based on electronic medical records. BMC bioinformatics 19(5):117. find
  16. TH Wen, D Vandyke, N Mrkšı́c, M Gašı́c, LM Rojas-Barahona, PH Su, S Ultes, and S Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017-Proceedings of Conference. volume… find
  17. David Milward and Martin Beveridge. 2003. Ontology-based dialogue systems. In Proc. 3rd Workshop on Knowledge and reasoning in practical dialogue systems (IJCAI03). pages 9–18. find
  18. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529–533. find
  19. Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building task-oriented dialogue systems for online shopping. In AAAI. pages 4618–4626. find
  20. Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Yun-Nung Chen, and Kam-Fai Wong. 2017a. Adversarial advantage actor-critic model for task-completion dialogue policy learning https://arxiv.org/abs/1710.11277. — [arXiv:1710.11277](https://arxiv.org/abs/1710.11277) ↗
  21. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE 101(5):1160–1179. find
  22. Tao Zheng, Wei Xie, Liling Xu, Xiaoying He, Ya Zhang, Mingrong You, Gong Yang, and You Chen. 2017. A machine learning-based framework to identify type 2 diabetes through electronic health records. International journal of medical informatics 97:120–127. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2018-task-oriented-dialogue-system-automatic/