Elamur Research
☰
home / papers / 2019 / BioBERT: a pre-trained biomedical language representation model for biomedical text mining

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1901.08746

Summary

Biomedical literature grows faster than researchers can read it, so automatic text mining matters more each year. General-purpose language models such as BERT are trained on Wikipedia and books, and they struggle with biomedical text, where vocabulary and word statistics are very different. The authors built BioBERT to address this. They started from the original BERT weights and continued pre-training on large biomedical collections: PubMed abstracts and PubMed Central (PMC) full-text articles. They kept BERT's architecture and its cased WordPiece vocabulary, which keeps BioBERT compatible with existing BERT tooling. They then fine-tuned the model, with only minimal task-specific changes, for three tasks: named entity recognition (NER), relation extraction (RE) and question answering (QA). BioBERT clearly beat both vanilla BERT and earlier state-of-the-art systems on all three tasks, and the biggest gain was in QA. The weights and fine-tuning code are public, which made BioBERT a standard baseline for domain adaptation. It also gives a template for adapting BERT-style models to other specialised domains or languages.

Key points

  • Pre-training corpora: English Wikipedia (2.5B words) and BooksCorpus (0.8B) from the original BERT, plus PubMed abstracts (4.5B) and PMC full texts (13.5B).
  • Continued pre-training took 23 days on eight NVIDIA V100 GPUs. The original BERT BASE vocabulary was kept instead of building a new biomedical one.
  • Improvements over the previous state of the art: +0.62 F1 on biomedical NER, +2.80 F1 on RE and +12.24 MRR on QA (BioASQ-style).
  • RE is handled as sentence classification over the [CLS] token, with target entities masked by placeholder tags such as @GENE$ and @DISEASE$. NER uses token-level BIO2 tagging.
  • The weights are released at github.com/naver/biobert-pretrained, and the fine-tuning code at github.com/dmis-lab/biobert.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

1120 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 1091 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1093/bioinformatics/btz682 · arXiv 1901.08746

@misc{lee2019biobert,
  title = {{BioBERT: a pre-trained biomedical language representation model for biomedical text mining}},
  author = {Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},
  year = {2019},
  doi = {10.1093/bioinformatics/btz682},
  eprint = {1901.08746},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1093/bioinformatics/btz682}
}

References

39 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alsentzer,E. et al. (2019) Publicly available clinical bert embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA. pp. 72–78. Association for Computational Linguistics. https://www.acl web.org/anthology/W19-1909. — [ACL W19-1909](https://aclanthology.org/W19-1909/) ↗
  2. Bhasuran,B. and Natarajan,J. (2018) Automatic extraction of gene-disease associations from literature using joint ensemble learning. PLoS One, 13, e0200699. find
  3. Bravo,À. et al. (2015) Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC Bioinformatics, 16, 55. find
  4. Devlin,J. et al. (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis,… find
  5. Do gan,R.I. et al. (2014) NCBI disease corpus: a resource for disease name recognition and concept normalization. J. Biomed. Inform., 47, 1–10. find
  6. Gerner,M. et al. (2010) Linnaeus: a species name identification system for biomedical literature. BMC Bioinformatics, 11, 85. find
  7. Giorgi,J.M. and Bader,G.D. (2018) Transfer learning for biomedical named entity recognition with neural networks. Bioinformatics, 34, 4087. find
  8. Habibi,M. et al. (2017) Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33, i37–i48. find
  9. Kim,J.-D. et al. (2004) Introduction to the bio-entity recognition task at JNLPBA. In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), Geneva, Switzerland. pp. 73–78. COLING. https://www.aclweb.org/anthology/W04-1213. ↗ our card
  10. Krallinger,M. et al. (2015) The chemdner corpus of chemicals and drugs and its annotation principles. J. Cheminform., 7. find
  11. Krallinger,M. et al. (2017) Overview of the BioCreative VI chemical-protein interaction track. In: Proceedings of the BioCreative VI Workshop, Bethesda, MD, USA, pp. 141–146. https://academic.oup.com/database/art icle/doi/10.1093/database/bay073/5055578. — [doi:10.1093/database/bay073/5055578](https://doi.org/10.1093/database/bay073/5055578) ↗
  12. Li,J. et al. (2016) Biocreative V CDR task corpus: a resource for chemical disease relation extraction. find
  13. Lim,S. and Kang,J. (2018) Chemical–gene relation extraction using recursive neural network. find
  14. Lin,C. et al. (2019) A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA. pp. 65–71. Association for Computational Linguistics. https://www. aclweb.org/an… find
  15. Lou,Y. et al. (2017) A transition-based joint model for disease named entity recognition and normalization. Bioinformatics, 33, 2363–2371. find
  16. Luo,L. et al. (2018) An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics, 34, 1381–1388. find
  17. McCann,B. et al. (2017) Learned in translation: contextualized word vectors. In: Guyon,I. et al. (eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pp. 6294–6305. http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf. ↗ our card
  18. Mikolov,T. et al. (2013) Distributed representations of words and phrases and their compositionality. In: Burges,C.J.C. (eds.), Advances in Neural Information Processing Systems 26, Curran Associates, Inc., pp. 3111–3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phra… ↗
  19. Mohan,S. and Li,D. (2019) Medmentions: a large biomedical corpus annotated with UMLS concepts. arXiv preprint arXiv: 1902.09476. — [arXiv:1902.09476](https://arxiv.org/abs/1902.09476) find
  20. Pafilis,E. et al. (2013) The species and organisms resources for fast and accurate identification of taxonomic names in text. PLoS One, 8, e65390. find
  21. Pennington,J. et al. (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. pp. 1532–1543. Association for Computational Linguistics. https://www.aclweb.org/anthology/D14-1162. ↗
  22. Peters,M.E. et al. (2018) Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA. pp. 2227–2237. Association for Computationa… find
  23. Pyysalo,S. et al. (2013) Distributional semantics resources for biomedical text processing. In: Proceedings of the 5th International Symposium on Languages in Biology and Medicine, Tokyo, Japan, pp. 39–43. https://aca demic.oup.com/bioinformatics/article/33/14/i37/3953940. find
  24. Rajpurkar,P. et al. (2016) Squad: 100,000þ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX. pp. 2383–2392. Association for Computational Linguistics. https://www.aclweb.org/anthol ogy/D16-1264. ↗ our card
  25. Sachan,D.S. et al. (2018) Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. In: Finale,D.-V. et al. (eds.), Proceedings of Machine Learning Research, Palo Alto, CA, Vol. 85, pp. 383–402. PMLR. http://proceedings.mlr.press/v85/sachan18a.htm… ↗
  26. Smith,L. et al. (2008) Overview of biocreative ii gene mention recognition. Genome Biol., 9, S2. find
  27. Sousa,D. et al. (2019) A silver standard corpus of human phenotype-gene relations. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN. pp. 1487–1492. Ass… find
  28. Sung,N. et al. (2017) NSML: A machine learning platform that enables you to focus on your models. arXiv preprint arXiv: 1712.05902. find
  29. Tsatsaronis,G. et al. (2015) An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16, 138. find
  30. Uzuner,Ö. et al. (2011) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J. Am. Med. Inform. Assoc., 18, 552–556. find
  31. Van Mulligen,E.M. et al. (2012) The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships. J. Biomed. Inform., 45, 879–884. find
  32. Vaswani,A. et al. (2017) Attention is all you need. In: Guyon,I. et al. (eds.), Advances in Neural Information Processing Systems, pp. 5998–6008. Curran Associates, Inc. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. ↗
  33. Wang,X. et al. (2018) Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35, 1745–1752. find
  34. Wiese,G. et al. (2017) Neural domain adaptation for biomedical question answering. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada. pp. 281–289. Association for Computational Linguistics. https://www.aclweb. org/anthology/K17-1029. ↗
  35. Wu,Y. et al. (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08144. find
  36. Xu,K. et al. (2019) Document-level attention-based BiLSTM-CRF incorporating disease dictionary for disease named entity recognition. Comput. Biol. Med., 108, 122–132. find
  37. Yoon,W. et al. (2019) Collabonet: collaboration of deep neural networks for biomedical named entity recognition. BMC Bioinformatics, 20, 249. find
  38. Zhu,H. et al. (2018) Clinical concept extraction with contextual word embedding. NIPS Machine Learning for Health Workshop. http://par.nsf.gov/bib lio/10098080. ↗
  39. Downloaded from https://academic.oup.com/bioinformatics/advance-article-abstract/doi/10.1093/bioinformatics/btz682/5566506 by guest on 18 October 2019 — [doi:10.1093/bioinformatics/btz682/5566506](https://doi.org/10.1093/bioinformatics/btz682/5566506) ↗