Elamur Research
☰
home / papers / 2019 / ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.05342

Summary

Free-text clinical notes describe patients more fully than structured EHR fields, but they are long, sparse, and full of jargon and abbreviations, so they are rarely used in predictive models. The authors build ClinicalBERT by taking BERT, originally trained on BooksCorpus and Wikipedia, continuing its pre-training on clinical notes from MIMIC-III, and then fine-tuning it to estimate a patient's risk of returning to hospital within 30 days. The model can score risk from a discharge summary or from notes written early in an ICU stay, updating as new notes arrive.

Evaluation emphasises a clinically motivated criterion: recall measured at a fixed high precision, chosen to avoid alarm fatigue. ClinicalBERT matches physician judgements of medical concept similarity better than Word2Vec and FastText. It also beats baseline models at readmission prediction, and its advantage grows as more notes become available. Its self-attention weights can be visualised to show which terms drive a prediction. The released weights and scripts give a reusable starting point for other note-based tasks, such as mortality or length-of-stay prediction.

Key points

  • ClinicalBERT is BERT further pre-trained on MIMIC-III notes, using masked language modeling and next sentence prediction. It is then fine-tuned with a linear layer on the [CLS] representation to predict 30-day readmission.
  • Readmission is predicted from discharge summaries and also from notes available 24–48h or 48–72h after admission. Patients discharged before the cutoff are excluded from evaluation. Performance is reported with AUROC, AUPRC and RP80.
  • Embedding quality is compared with Word2Vec and FastText by checking agreement with physician-rated similarity between medical concepts.
  • The encoder has 144 self-attention mechanisms (12 heads in each of 12 layers). Their weights highlight predictive words such as "chronic" and "acute".
  • Model parameters and code are released at github.com/kexinhuang12345/clinicalBERT. The authors advise re-training on local hospital EHR data, because MIMIC-III comes from the ICUs of a single Boston hospital.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

199 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 192 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.05342 · arXiv 1904.05342

@misc{huang2019clinicalbert,
  title = {{ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission}},
  author = {Huang, Kexin and Altosaar, Jaan and Ranganath, Rajesh},
  year = {2019},
  doi = {10.48550/arxiv.1904.05342},
  eprint = {1904.05342},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.05342}
}

References

40 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng, D. Jin, T. Naumann, and M. B. A. McDermott. “Publicly Available Clinical BERT Embeddings”. In: arXiv:1904.03323 (2019). find our card
  2. G. F. Anderson and E. P. Steinberg. “Hospital readmissions in the Medicare population”. In: New England Journal of Medicine 21 (1984). find
  3. D. Banerjee, C. Thompson, C. Kell, R. Shetty, Y. Vetteth, H. Grossman, A. DiBiase, and M. Fowler. “An informatics-based approach to reducing heart failure all-cause readmissions: the Stanford heart failure dashboard”. In: Journal of the American Medical Informatics Association 3 (2016). find
  4. S. Basu Roy, A. Teredesai, K. Zolfaghar, R. Liu, D. Hazel, S. Newman, and A. Marinez. “Dynamic Hierarchical Classification for Patient Risk-of-Readmission”. In: Knowledge Discovery and Data Mining (2015). find
  5. W. Boag, D. Doss, T. Naumann, and P. Szolovits. “What’s in a Note? Unpacking Predictive Value in Clinical Note Representations”. In: AMIA Joint Summits on Translational Science (2018). find
  6. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. “Enriching word vectors with subword information”. In: Transactions of the Association for Computational Linguistics (2017). find
  7. X. Cai, O. Perez-Concha, E. Coiera, F. Martin-Sanchez, R. Day, D. Roffe, and B. Gallego. “Real-time prediction of mortality, readmission, and length of stay using electronic health record data”. In: Journal of the American Medical Informatics Association 3 (2015). find
  8. R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. “Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission”. In: Knowledge Discovery and Data Mining. 2015. find
  9. W. W. Chapman, P. M. Nadkarni, L. Hirschman, L. W. D’Avolio, G. K. Savova, and O. Uzuner. “Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions”. In: Journal of the American Medical Informatics Association 5 (2011). find
  10. B. Chiu, G. Crichton, A. Korhonen, and S. Pyysalo. “How to Train good Word Embeddings for Biomedical NLP”. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing, ACL 2016 (). find
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: arXiv:1810.04805 (2018). find
  12. J. Futoma, J. Morris, and J. Lucas. “A comparison of models for predicting early hospital readmissions”. In: Journal of Biomedical Informatics (2015). find
  13. B. A. Goldstein, A. M. Navar, M. J. Pencina, and J. P. A. Ioannidis. “Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review”. In: Journal of the American Medical Informatics Association (2017). find
  14. S. Hochreiter and J. Schmidhuber. “Long Short-Term Memory”. In: Neural Computation 8 (1997). find
  15. A. E. W. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. “MIMIC-III, a freely accessible critical care database”. In: Scientific Data (2016). find our card
  16. A. J. Kind and M. A. Smith. “Documentation of mandated discharge summary components in transitions from acute to subacute care”. In: Agency for Healthcare Research and Quality (2008). find
  17. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang. “BioBERT: a pre-trained biomedical language representation model for biomedical text mining”. In: arXiv:1901.08746 (2019). find our card
  18. J. Liu, Z. Zhang, and N. Razavian. “Deep EHR: Chronic Disease Prediction Using Medical Notes”. In: Proceedings of the 3rd Machine Learning for Healthcare Conference. 2018. find
  19. L. van der Maaten and G. Hinton. “Visualizing data using t-SNE”. In: Journal of Machine Learning Research (2008). find
  20. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. “Distributed representations of words and phrases and their compositionality”. In: Advances in Neural Information Processing Systems. 2013. find
  21. C. A. Pedersen, P. J. Schneider, and D. J. Scheckelhoff. “ASHP national survey of pharmacy practice in hospital settings: Prescribing and transcribing—2016”. In: American Journal of Health-System Pharmacy 17 (2017). find
  22. T. Pedersen, S. V. Pakhomov, S. Patwardhan, and C. G. Chute. “Measures of semantic similarity and relatedness in the biomedical domain”. In: Journal of Biomedical Informatics 3 (2007). find
  23. J. Pennington, R. Socher, and C. Manning. “Glove: Global Vectors for Word Representation”. In: EMNLP (2014). find
  24. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. “Deep contextualized word representations”. In: arXiv:1802.05365 (2018). find
  25. A. Radford. “Improving Language Understanding by Generative Pre-Training”. https : / / s3 - us - west - 2 . amazonaws . com/openai-assets/research-covers/language-unsupervised/ language\_understanding\_paper.pdf. 2018. find
  26. A. Rajkomar, E. Oren, K. Chen, A. M. Dai, N. Hajaj, M. Hardt, P. J. Liu, X. Liu, J. Marcus, M. Sun, P. Sundberg, H. Yee, K. Zhang, Y. Zhang, G. Flores, G. E. Duggan, J. Irvine, Q. Le, K. Litsch, A. Mossin, J. Tansuwan, D. Wang, J. Wexler, J. Wilson, D. Ludwig, S. L. Volchenboum, K. Chou, M. Pearson,… find
  27. M. Schuster and K. K. Paliwal. “Bidirectional recurrent neural networks”. In: IEEE Trans. Signal Processing (1997). find
  28. S. Sendelbach and M. Funk. “Alarm fatigue: a patient safety concern”. In: AACN Advanced Critical Care 4 (2013). find
  29. R. Sennrich, B. Haddow, and A. Birch. “Neural Machine Translation of Rare Words with Subword Units”. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016. find
  30. B. Shickel, P. J. Tighe, A. Bihorac, and P. Rashidi. “Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis”. In: IEEE Journal of Biomedical and Health Informatics 5 (2018). find
  31. Y. Si, J. Wang, H. Xu, and K. Roberts. “Enhancing clinical concept extraction with contextual embeddings”. In: Journal of the American Medical Informatics Association 11 (2019). find
  32. C. Van Walraven, R. Seth, P. C. Austin, and A. Laupacis. “Effect of discharge summary availability during post-discharge visits on hospital readmission”. In: Journal of General Internal Medicine 3 (2002). find
  33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention is all you need”. In: Advances in Neural Information Processing Systems. 2017. find
  34. Y. Wang, S. Liu, N. Afzal, M. Rastegar-Mojarad, L. Wang, F. Shen, P. Kingsbury, and H. Liu. “A comparison of word embeddings for the biomedical natural language processing”. In: Journal of Biomedical Informatics (2018). find
  35. W.-H. Weng, K. B. Wagholikar, A. T. McCray, P. Szolovits, and H. C. Chueh. “Medical Subdomain Classification of Clinical Notes Using a Machine Learning-Based Natural Language Processing Approach”. In: BMC Medical Informatics and Decision Making 1 (2017). find
  36. C. Xiao, E. Choi, and J. Sun. “Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review”. In: Journal of the American Medical Informatics Association 10 (2018). find
  37. K.-H. Yu, A. L. Beam, and I. S. Kohane. “Artificial intelligence in healthcare”. In: Nature Biomedical Engineering 10 (2018). find
  38. Y. Zhang, R. Jin, and Z.-H. Zhou. “Understanding bag-of-words model: a statistical framework”. In: International Journal of Machine Learning and Cybernetics 1 (2010). find
  39. Y. Zhang, R. Henao, Z. Gan, Y. Li, and L. Carin. “Multi-Label Learning from Medical Plain Text with Convolutional Residual Models”. In: Proceedings of the 3rd Machine Learning for Healthcare Conference. 2018. find
  40. R. B. Zuckerman, S. H. Sheingold, E. J. Orav, J. Ruhter, and A. M. Epstein. “Readmissions, observation, and the hospital readmissions reduction program”. In: New England Journal of Medicine 16 (2016). find