Elamur Research
☰
home / papers / 2019 / Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5006.pdf

Summary

Biomedical language representation models such as BioBERT or clinical BERT were hard to compare, because each study reported results on its own choice of datasets or on differently sized versions of the same ones. The authors modeled their solution on GLUE and built BLUE (Biomedical Language Understanding Evaluation). This benchmark groups ten established shared-task corpora into five task types: sentence similarity, named entity recognition, relation extraction, multilabel document classification and inference. The corpora cover both research literature and clinical notes and differ in size and difficulty. As baselines, the authors fine-tune BERT and ELMo models. For BERT, they pre-train their own Base and Large versions either on PubMed abstracts alone or on PubMed plus MIMIC-III notes. Overall, BERT-Base trained on the combined PubMed+MIMIC-III data performs best, with a clear lead on clinical tasks. This suggests that the genre of the pre-training text matters. The authors release the data, preprocessing scripts, evaluation code and pre-trained weights, which gives the field a common yardstick.

Key points

  • Ten corpora and their metrics:
    • MedSTS and BIOSSES: Pearson.
    • BC5CDR-disease, BC5CDR-chemical and ShARe/CLEFE: strict F1.
    • DDI, ChemProt and i2b2 2010: F1, micro-averaged for DDI and ChemProt.
    • HoC: example-based F1.
    • MedNLI: accuracy.
  • Four BERT variants were pre-trained: BERT-Base (P), BERT-Large (P), BERT-Base (P+M) and BERT-Large (P+M). P stands for PubMed and M for MIMIC-III.
  • Base vs. Large:
    • Base usually beat Large. The exceptions were relation extraction and document classification, which have longer average sentences.
    • BERT-Large (P) was best on HoC.
    • BERT-Large (P+M) was weakest overall, which the authors attribute to the relatively small size of MIMIC-III.
  • Dataset sizes vary widely: BIOSSES has only 64 training and 16 test pairs, which made results unstable, while MedNLI has 11,232 training pairs. Relation extraction was handled as sentence classification by replacing the entity mentions with tags such as @GENE$.
  • Resources are public at github.com/ncbi-nlp/BLUE Benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

200 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 198 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5006

@inproceedings{peng2019transfer,
  title = {{Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets}},
  author = {Peng, Yifan and Yan, Shankai and Lu, Zhiyong},
  booktitle = {Proceedings of the 18th BioNLP Workshop and Shared Task},
  year = {2019},
  doi = {10.18653/v1/w19-5006},
  url = {https://doi.org/10.18653/v1/w19-5006}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. a freely accessible critical care database. Scientific data, 3:1600… find
  2. Daniel Jurafsky and James H. Martin. 2008. Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2 edition. Prentice Hall. find
  3. Bryan Rink, Sanda Harabagiu, and Kirk Roberts. 2011. Automatic extraction of relations between medical concepts in clinical texts. Journal of the American Medical Informatics Association, 18:594–600. find
  4. Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martı́n Pérez Pérez, Jesús Santamarı́a, Gael Pérez Rodrı́guez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López1 Umesh Nandal, Erin Van Buel, Akileshwari Chandrasekhar, Marleen Rodenburg, Astrid Laegreid, Marius Doornenbal, Julen… find
  5. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of EMNLP, pages 1586– 1596. find our card
  6. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing clinical concept extraction with contextual embedding. arXiv preprint arXiv:1902.08691. — [arXiv:1902.08691](https://arxiv.org/abs/1902.08691) find
  7. Neil R Smalheiser, Aaron M Cohen, and Gary Boni-field. 2019. Unsupervised low-dimensional vector representations for words, phrases and text that are transparent, scalable, and produce similarity metrics that are not redundant with neural embeddings. Journal of biomedical informatics, 90:103096. find
  8. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260–270. find
  9. Robert Leaman, Ritu Khare, and Zhiyong Lu. 2015. Challenges in clinical natural language processing for automated disorder normalization. Journal of biomedical informatics, 57:28–37. find
  10. Gizem Soğancıoğlu, Hakime Öztürk, and Arzucan Özgür. 2017. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics (Oxford, England), 33:i49–i58. find
  11. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. arXiv:1901.08746. find our card
  12. Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. 2013. Overview of the ShARe/CLEF eHealth evaluation lab 2013. In International Conference of the Cross-Language Evaluatio… find
  13. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database: the journal of biological databases and… find
  14. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association (JAMIA), 18:552–556. find
  15. Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. In Proceedings of NAACL-HLT, pages 1030– 1040. find
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  17. Yifan Peng, Anthony Rios, Ramakanth Kavuluru, and Zhiyong Lu. 2018. Extracting chemical-protein relations with ensembles of SVM and deep learning models. Database: the journal of biological databases and curation, 2018. find
  18. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  19. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532–1543. find
  20. Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2018b. MedSTS: a resource for clinical semantic textual similarity. Language Resources and Evaluation, pages 1–16. find
  21. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of ACL, pages 1756–1765. find our card
  22. Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018c. A comparison of word embeddings for the biomedical natural language processing. Journal of biomedical informatics, 87:12–20. find
  23. Shankai Yan and Ka-Chun Wong. 2017. Elucidating high-dimensional cancer hallmark annotation via enriched ontology. Journal of biomedical informatics, 73:84–94. find
  24. Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang. 2018. CollaboNet: collaboration of deep neural networks for biomedical named entity recognition. arXiv preprint arXiv:1809.07950. — [arXiv:1809.07950](https://arxiv.org/abs/1809.07950) find
  25. Min-Ling Zhang and Zhi-Hua Zhou. 2014. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819–1837. find