Elamur Research
☰
home / papers / 2019 / Publicly Available Clinical BERT Embeddings

Publicly Available Clinical BERT Embeddings

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.03323

Summary

Contextual encoders like BERT had transformed general-domain NLP, but no pretrained BERT for clinical narratives (e.g., physician notes) was publicly available. Such text differs noticeably from both web prose and biomedical literature. The authors pretrained BERT variants on roughly 2 million notes from MIMIC-III v1.4. They built one model from all note types and one from discharge summaries alone. Each was initialized either from BERT-Base or from BioBERT. Training followed the standard recipe, and evaluation used only a single linear layer on top of the fine-tuned encoder. Across MedNLI and four i2b2 tasks, the clinical models beat general BERT and BioBERT on MedNLI, i2b2 2010 and i2b2 2012. MedNLI accuracy reached a new state of the art. On the two de-identification tasks, the clinical models showed no gain. The authors attribute this to a distribution mismatch: MIMIC replaces PHI with placeholder tokens, whereas the de-ID corpora insert realistic synthetic surrogates. The main value of the work is the free release of the models, which spares other labs weeks of GPU time.

Key points

  • Four released checkpoints are available at github.com/EmilyAlsentzer/clinicalBERT: Clinical BERT, Discharge Summary BERT, Bio+Clinical BERT and Bio+Discharge Summary BERT.
  • On MedNLI, the Bio+Clinical variants scored 82.7% accuracy. This compares with 77.6% for BERT, 80.8% for BioBERT and the earlier best of 73.5% from InferSent.
  • On i2b2 2010 concept extraction, Bio+Discharge Summary BERT reached an exact F1 of 87.8. BioBERT scored 86.5 and general BERT 83.5.
  • On the de-ID tasks (i2b2 2006, i2b2 2014), general BERT or BioBERT matched or outperformed the clinical models. For example, on i2b2 2006, Clinical BERT scored 91.5 against 94.8 for BioBERT.
  • Pretraining took about 17–18 days on a single GTX TITAN X GPU.
  • Stated limitations:
    • The task heads are simple.
    • The notes come only from the ICU of one hospital (BIDMC).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

512 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 505 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.03323 · arXiv 1904.03323

@misc{alsentzer2019publicly,
  title = {{Publicly Available Clinical BERT Embeddings}},
  author = {Alsentzer, Emily and Murphy, John R. and Boag, Willie and Weng, Wei-Hung and Jin, Di and Naumann, Tristan and McDermott, Matthew B. A.},
  year = {2019},
  doi = {10.48550/arxiv.1904.03323},
  eprint = {1904.03323},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.03323}
}

References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  2. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. arXiv:1705.02364 [cs]. ArXiv: 1705.02364. find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. find
  4. Noemie Elhadad, Sameer Pradhan, Sharon Gorman, Suresh Manandhar, Wendy Chapman, and Guergana Savova. SemEval-2015 Task 14: Analysis of Clinical Text. page 8. find
  5. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. arXiv:1801.06146 [cs, stat]. ArXiv: 1801.06146. find
  6. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3:160035. find our card
  7. Kaung Khin, Philipp Burckhardt, and Rema Padman. 2018. A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation. arXiv:1810.01570 [cs]. ArXiv: 1810.01570. — [arXiv:1810.01570](https://arxiv.org/abs/1810.01570) find
  8. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. arXiv:1901.08746 [cs]. ArXiv: 1901.08746. find our card
  9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  10. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. arXiv:1902.07669 [cs]. ArXiv: 1902.07669. find
  11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  12. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365 [cs]. ArXiv: 1802.05365. find
  13. Sameer Pradhan, Nomie Elhadad, Wendy Chapman, Suresh Manandhar, and Guergana Savova. 2014. SemEval-2014 Task 7: Analysis of Clinical Text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62, Dublin, Ireland. Association for Computational Linguistics a… find
  14. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text Chunking using Transformation-Based Learning. arXiv:cmp-lg/9505040. ArXiv: cmp-lg/9505040. find
  15. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from Natural Language Inference in the Clinical Domain. arXiv:1808.06752 [cs]. ArXiv: 1808.06752. find our card
  16. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing Clinical Concept Extraction with Contextual Embedding. arXiv:1902.08691 [cs]. ArXiv: 1902.08691. — [arXiv:1902.08691](https://arxiv.org/abs/1902.08691) find
  17. Amber Stubbs, Christopher Kotfila, and zlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1. Journal of Biomedical Informatics, 58 Suppl:S11–19. find
  18. Amber Stubbs and zlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics, 58 Suppl:S20–29. find
  19. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013a. Annotating temporal information in clinical narratives. Journal of Biomedical Informatics, 46 Suppl:S5–12. find
  20. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013b. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. Journal of the American Medical Informatics Association: JAMIA, 20(5):806–813. find
  21. Ozlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association: JAMIA, 14(5):550–563. zlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, as… find
  22. Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical Concept Extraction with Contextual Word Embedding. arXiv:1810.10566 [cs]. ArXiv: 1810.10566. — [arXiv:1810.10566](https://arxiv.org/abs/1810.10566) find