Elamur Research⌕
☰
home / papers / 2020 / Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art

Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art

Summary

Many pretrained models exist for biomedical and clinical NLP, but each has been tested on different tasks with different splits and metrics. This makes it hard to choose between a strong general-purpose model and an older domain-adapted one. The authors built a common testbed of 18 established tasks covering PubMed and clinical text. It includes sequence labelling (NER, de-identification) and classification (relation extraction, multi-label classification, NLI). On this testbed they compared SciBERT, BioBERT, ClinicalBERT, BioMed-RoBERTa and general RoBERTa checkpoints under one fine-tuning setup.

BioBERT came out ahead on biomedical tasks, while the general RoBERTa-large did best on clinical ones. The authors then pretrained their own RoBERTa-style models with FAIRSEQ on public corpora: PubMed abstracts, PubMed Central and MIMIC-III. Through ablations they varied model size, vocabulary and corpus. The strongest configuration combined:

  • a large model,
  • an in-domain vocabulary,
  • mixed biomedical and clinical data.

This configuration set new state-of-the-art results on 9 tasks. A distilled base model also beat comparably sized competitors. The models and code are publicly released, which gives practitioners a tested default.

Key points

  • The benchmark has 18 tasks. Examples include BC5-CDR, JNLPBA, NCBI-Disease, BC4CHEMD, BC2GM, LINNAEUS, Species-800, I2B2-2010/2012/2014, HOC, ChemProt, GAD, EU-ADR, DDI-2013, I2B2-2010-RE and MedNLI. Tasks were chosen to overlap with BLUE, BioBERT, SciBERT and ClinicalBERT evaluations.
  • The pretraining corpora are all freely available:
    • PubMed abstracts: 27GB, 22M abstracts, about 4.2B words.
    • PMC full texts: 60GB, 3.4M articles, about 9.6B words.
    • MIMIC-III physician notes: 3.3GB, about 0.5B words.
  • The domain vocabulary is a 50k-unit byte-level BPE learned on PubMed, the same size as RoBERTa's. Models with this vocabulary were trained from random initialization; the others started from RoBERTa checkpoints.
  • Distillation (teacher: large model, student: base model) ran for 50k steps. The loss combined three weighted terms:
    • MLM loss (weight 5.0),
    • KL divergence with temperature 2.0 (weight 2.0),
    • an optional cosine loss aligning hidden states (weight 1.0).
  • The best student improved on the base model by about 0.3 points on average but stayed roughly 0.8 points behind the teacher. Models and code are at github.com/facebookresearch/bio-lm.

Interesting ideas

  • I1. Evaluate every model on one shared testbed. Rerunning all candidates under one shared fine-tuning protocol removes confusion caused by mismatched splits and metrics. This is a simple way to make a fair model choice for any under-resourced language or domain.
  • I2. Report results by subgroup. Separate averages for clinical vs. PubMed tasks and for sequence labelling vs. classification revealed that the "best" model depends on the subdomain. The same breakdown would help any multi-domain benchmark avoid hiding trade-offs behind one number.
  • I3. Treat vocabulary as a deliberate design choice. The authors accepted the cost of training from scratch to get an in-domain tokenizer, and it paid off. This suggests that domains or languages poorly covered by a general tokenizer may benefit from a dedicated one rather than only continued pretraining.
  • I4. Use only openly licensed corpora. Restricting pretraining data to freely licensed sources keeps the resulting models releasable, which matters whenever the goal is a shared community resource.
  • I5. Add a "train longer" control for distillation. Comparing the distilled student against the same student simply trained for the extra steps separates the real effect of distillation from the effect of extra compute. This is a cheap control worth copying in any compression experiment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Life sciences & medicine

Cited by

73 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.clinicalnlp-1.17

@inproceedings{lewis2020pretrained,
  title = {{Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art}},
  author = {Lewis, Patrick and Ott, Myle and Du, Jingfei and Stoyanov, Veselin},
  booktitle = {Proceedings of the 3rd Clinical Natural Language Processing Workshop},
  year = {2020},
  doi = {10.18653/v1/2020.clinicalnlp-1.17},
  url = {https://doi.org/10.18653/v1/2020.clinicalnlp-1.17}
}

References

59 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computati… find our card
  2. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Wilhelm, Zhen… find our card
  3. Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Johan Högberg, Ulla Stenius, and Anna Korhonen. 2016. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics, 32(3):432–440. Publisher: Oxford Academic. find
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620,… find
  5. Balu Bhasuran, Jeyakumar Natarajan, and . . 2018. Automatic extraction of gene-disease associations from literature using joint ensemble learning. PloS One, 13(7):e0200699. find
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching Word Vectors with Subword Information. arXiv:1607.04606 [cs]. ArXiv: 1607.04606. find
  7. Àlex Bravo, Janet Piñero, Núria Queralt-Rosinach, Michael Rautschka, and Laura I. Furlong. 2015. Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC bioinformatics, 16:55. find
  8. Tom B. Brown, Benjamin Pickman Mann, Nick Ryder, Melanie Subbiah, Jean Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  9. Billy Chiu, Gamal Crichton, Anna Korhonen, and Sampo Pyysalo. 2016. How to Train good Word Embeddings for Biomedical NLP. In Proceedings of the 15th Workshop on Biomedical Natural Language Processing, pages 166–174, Berlin, Germany. Association for Computational Linguistics. find
  10. Nigel Collier and Jin-Dong Kim. 2004. Introduction to the Bio-entity Recognition Task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73–78, Geneva, Switzerland. COLING. find our card
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  13. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. NCBI Disease Corpus: A Resource for Disease Name Recognition and Concept Normalization. Journal of biomedical informatics, 47:1–10. find
  14. Martin Gerner, Goran Nenadic, and Casey M. Bergman. 2010. LINNAEUS: a species name identification system for biomedical literature. BMC bioinformatics, 11:85. find
  15. John M. Giorgi and Gary D. Bader. 2018. Transfer learning for biomedical named entity recognition with neural networks. Bioinformatics (Oxford, England), 34(23):4087–4094. find
  16. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. arXiv:2007.15779 [cs]. ArXiv: 2007.15779. find our card
  17. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. arXiv:2004.10964 [cs]. ArXiv: 2004.10964. find
  18. Marı́a Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martı́nez, and Thierry Declerck. 2013. The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactions. Journal of Biomedical Informatics, 46(5):914–920. and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource f… find
  19. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. find our card
  20. Martin Krallinger, Obdulia Rabal, Saber Ahmad Akhondi, Martı́n Pérez Pérez, Jésús López Santa-marı́a, Gael Pérez Rodrı́guez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio Baso López, Umesh Nandal, Erin M. van Buel, A. Poorna Chandrasekhar, Marleen Rodenburg, Astrid Lægreid, Marius A… find
  21. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M. Lowe, Roger A. Sayle, Riza Theresa Batista-Navarro, Rafal Rak, Torsten Huber, Tim Rocktäschel, Sérgio Matos, David Campos, Buzhou Tang, Hua Xu, Tsendsuren M… find
  22. Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. arXiv:1804.10959 [cs]. ArXiv: 1804.10959. find
  23. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv:1901.07291 [cs]. ArXiv: 1901.07291. find
  24. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. eprint: https://academic.oup.com/bioinformatics/article-pdf/36/4/1234/325… ↗ our card
  25. Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y.-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2019. Don’t Say That! Making Inconsistent Dial… find our card
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  27. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  28. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pages 3111–3119, USA. Curran A… find
  29. Erik M. van Mulligen, Annie Fourrier-Reglat, David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A. Kors, and Laura I. Furlong. 2012. The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships. Journal of Biomedical Informatics, 45(5):879–884. find
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  31. Evangelos Pafilis, Sune P. Frankild, Lucia Fanini, Sarah Faulwetter, Christina Pavloudi, Aikaterini Vasileiadou, Christos Arvanitidis, and Lars Juhl Jensen. 2013. The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text. PloS One, 8(6):e65390. find
  32. Yifan Peng, Qingyu Chen, and Zhiyong Lu. 2020. An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining. arXiv:2005.02799 [cs]. ArXiv: 2005.02799. find
  33. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58– 65, Florence, Italy. Association for Computational Linguistic… find our card
  34. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  35. Amber Stubbs and Ozlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: the 2014 i2b2/UTHealth Corpus. Journal of biomedical informatics, 58(Suppl):S20–S29. find
  36. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365 [cs]. ArXiv: 1802.05365. find
  37. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013a. Annotating temporal information in clinical narratives. Journal of Biomedical Informatics, 46 Suppl:S5–12. find
  38. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2019. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. arXiv:1811.01088 [cs]. ArXiv: 1811.01088. find our card
  39. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013b. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. Journal of the American Medical Informatics Association : JAMIA, 20(5):806– 813. find
  40. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  41. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from Natural Language Inference in the Clinical Domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics. find our card
  42. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs]. ArXiv: 1910.01108. find
  43. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  44. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing Clinical Concept Extraction with Contextual Embeddings. Journal of the American Medical Informatics Association, 26(11):1297–1304. ArXiv: 1902.08691. — [arXiv:1902.08691](https://arxiv.org/abs/1902.08691) find
  45. Larry Smith, Lorraine K. Tanabe, Rie Johnson nee Ando, Cheng-Ju Kuo, I.-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M. Friedrich, Kuzman Ganchev, Manabu Torii, Hongfang Liu, Barry Haddow, Craig A. Struble, Richard J. Povinelli, Andreas Vlachos, William A. Baumgartner, Lawrence Hun… find
  46. Amber Stubbs, Christopher Kotfila, and Ozlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1. Journal of biomedical informatics, 58(Suppl):S11–S19. find
  47. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, Online. Association for Computatio… find our card
  48. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… find
  49. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. arXiv:1908.08962 [cs]. ArXiv: 1908.08962. find our card
  50. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association : JAMIA, 18(5):552–556. find
  51. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\\textquotesingle Alché-Buc, E. Fox, an… find our card
  52. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355… find our card
  53. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex Wade, Kuansan Wan… find our card
  54. Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018b. A comparison of word embeddings for the biomedical natural language processing. Journal of Biomedical Informatics, 87:12–20. find
  55. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  56. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. find our card
  57. Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang. 2019. CollaboNet: collaboration of deep neural networks for biomedical named entity recognition. BMC Bioinformatics, 20(10):249. find
  58. Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Scientific Data, 6(1):52. Number: 1 Publisher: Nature Publishing Group. find
  59. Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical Concept Extraction with Contextual Word Embedding. arXiv:1810.10566 [cs]. ArXiv: 1810.10566. — [arXiv:1810.10566](https://arxiv.org/abs/1810.10566) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lewis-2020-pretrained-language-models-biomedical-clinical/