Elamur Research
☰
home / papers / 2018 / Lessons from Natural Language Inference in the Clinical Domain

Lessons from Natural Language Inference in the Clinical Domain

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1187.pdf

Summary

Neural NLI models perform well on open-domain benchmarks such as SNLI, but they generalize poorly to specialized fields where labeled data is scarce and reasoning depends on expert knowledge. Romanov and Shivade address this for medicine by building MedNLI. Its premises are sentences drawn from the Past Medical History sections of MIMIC-III clinical notes. Following the SNLI protocol, clinicians wrote an entailed, a neutral and a contradicting hypothesis for each premise.

The authors benchmark several systems on the new dataset:

  • a gradient-boosting model with hand-crafted and UMLS-based features,
  • BOW,
  • InferSent,
  • ESIM.

They then test two ways to improve these baselines. The first is transfer learning from open-domain corpora. The second is injecting domain knowledge, either through embeddings trained on clinical text or through the SNOMED-CT/UMLS ontology (retrofitting and a proposed knowledge-directed attention). Both strategies yield gains, with domain-specific embeddings giving the clearest improvement. Ontology integration helps only modestly.

The work matters because it supplies the first expert-annotated clinical NLI benchmark with public baselines. It also shows that medical inference typically requires multi-step knowledge that open-domain models lack.

Key points

  • MedNLI size and split: four clinicians annotated 4,683 premises over six weeks, producing 14,049 sentence pairs. These are split 11,232 / 1,395 / 1,422 (train/dev/test), with no premise shared across splits.
  • Annotation quality: a pilot of 552 pairs between two radiologists reached Cohen's κ = 0.78. Most disagreements involved "definitely true" versus "maybe true." The final data is single-annotated and distributed via the MIMIC-III derived data repository.
  • Ontology methods: retrofitting embeddings with ontology relations lowered accuracy by about 1.7–2.7 points. Knowledge-directed attention gave small gains of up to 0.3 points, and about 60% of those gains came from the neutral class.
  • Path lengths: shortest UMLS paths between premise and hypothesis concepts are often long, unlike WordNet paths in SNLI. This points to multi-step reasoning in MedNLI.
  • Errors and artifacts: neutral is the hardest class. Typical errors involve numerical lab values, abbreviations, negation, and world or medical knowledge. A hypothesis-only classifier still reaches 61.9 F1, compared with 67.0 on SNLI and 53.9 on MultiNLI, which reveals annotation artifacts in MedNLI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1187

@inproceedings{romanov2018lessons,
  title = {{Lessons from Natural Language Inference in the Clinical Domain}},
  author = {Romanov, Alexey and Shivade, Chaitanya},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1187},
  url = {https://doi.org/10.18653/v1/d18-1187}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR. find our card
  2. Alan R Aronson and François-Michel Lang. 2010. An overview of metamap: historical perspective and recent advances. Journal of the American Medical Informatics Association, 17(3):229–236. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. find our card
  4. Olivier Bodenreider. 2004. The Unified Medical Language System (UMLS): Integrating biomedical terminology. Nucleic Acids Research, 32(suppl 1):D267–D270. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5. find
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. find
  7. Anita Burgun and Olivier Bodenreider. 2001. Comparing terms, concepts and semantic classes in Word-Net and the Unified Medical Language System. In Proceedings of the NAACL Workshop: WordNet and other lexical resources: Applications, extensions and customizations., pages 77–82. find
  8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Procedings of ACL. find
  9. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. find our card
  10. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP. find our card
  11. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. find
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
  13. Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of NAACL. find
  14. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  15. Carol Friedman, Pauline Kra, and Andrey Rzhetsky. 2002. Two biomedical sublanguages: a description based on the theories of Zellig Harris. Journal of Biomedical Informatics, 35(4):222–235. find
  16. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 315–323. find
  17. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. Proceedings of NAACL. find
  18. Paul KJ Han, William MP Klein, and Neeraj K Arora. 2011. Varieties of uncertainty in health care: a conceptual taxonomy. Medical Decision Making, 31(6):828–838. find
  19. Alistair EW Johnson, Tom J Pollard, Lu Shen, Liwei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3. find our card
  20. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  21. Ji Young Lee, Franck Dernoncourt, and Peter Szolovits. 2018. Transfer learning for named-entity recognition with neural networks. Proceedings of LREC. find
  22. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195. find
  23. Lingpipe. 2008. LingPipe 4.1.0. find
  24. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of NIPS. find our card
  25. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. find
  26. Stephane M. Meystre, Guergana K. Savova, Karin C. Kipper-Schuler, and John F. Hurdle. 2008. Extracting information from textual documents in the electronic health record: a review of recent research. Yearbook of Medical Informatics, 35:128–44. find
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  28. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of EMNLP. find our card
  29. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL. find
  30. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of EMNLP. find
  31. Ted Pedersen, Serguei VS Pakhomov, Siddharth Patwardhan, and Christopher G Chute. 2007. Measures of semantic similarity and relatedness in the biomedical domain. Journal of Biomedical Informatics, 40(3):288–299. find
  32. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP. find
  33. Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. 2017. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225. — [arXiv:1711.05225](https://arxiv.org/abs/1711.05225) find
  34. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of ICLR. find our card
  35. Sunil Kumar Sahu and Ashish Anand. 2017. What matters in a transferable neural network model for relation classification in the biomedical domain? arXiv preprint arXiv:1708.03446. — [arXiv:1708.03446](https://arxiv.org/abs/1708.03446) find
  36. Chaitanya Shivade, Courtney Hebert, Marcelo Lopetegui, Marie-Catherine De Marneffe, Eric Fosler-Lussier, and Albert M Lai. 2015. Textual inference for eligibility criteria resolution in clinical trials. Journal of Biomedical Informatics, 58:S211–S218. find
  37. Gabriel Stanovsky, Daniel Gruhl, and Pablo Mendes. 2017. Recognizing mentions of adverse drug reaction in social media using knowledge-infused recurrent models. In Proceedings of EACL. find
  38. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answeri… find
  39. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems, pages 2773–2781. find