Elamur Research
☰
home / papers / 2019 / SciBERT: A Pretrained Language Model for Scientific Text

SciBERT: A Pretrained Language Model for Scientific Text

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1371.pdf

Summary

Labeled data for scientific NLP is scarce and expensive, because annotation requires domain experts. Popular pretrained models such as BERT and ELMo learn only from general-domain text like news and Wikipedia. To close this gap, the authors built SciBERT. It keeps the BERT-Base architecture but is pretrained on the full text of papers from the Semantic Scholar corpus, mostly biomedical with some computer science. They also created SciVocab, a WordPiece vocabulary built from that scientific corpus. The model was tested on NER, PICO extraction, text classification, relation classification and dependency parsing, using biomedical, computer science and multi-domain datasets. Each task was run two ways: finetuning the whole model, and training BiLSTM-based models on top of frozen embeddings. SciBERT beat BERT-Base consistently and set new state-of-the-art results on several benchmarks. On some of them it outperformed BioBERT even though it saw far less biomedical text. Ablations show that most of the gain comes from in-domain pretraining data, not from the custom vocabulary. The released weights and code give researchers a reusable starting point for low-resource scientific text mining.

Key points

  • Corpus: 1.14M full-text Semantic Scholar papers (82% biomedical, 18% computer science), about 3.17B tokens, close to the size of BERT's original training data. Sentences were split with ScispaCy.
  • Vocabulary: SciVocab has 30K tokens, the same size as BERT's BaseVocab, and shares only 42% of its tokens with it. Using SciVocab added about +0.60 F1 on average. Four variants were trained (cased/uncased × BaseVocab/SciVocab) on a single 8-core TPU v3, taking about one week for the from-scratch models.
  • Gains over BERT-Base: on average +2.11 F1 with finetuning and +2.43 F1 with frozen embeddings. Finetuning beat frozen embeddings plus task-specific models by about +3.25 F1 for SciBERT.
  • New SOTA results: BC5CDR, ChemProt (83.64 vs. BioBERT's 76.68), EBM-NLP, ACL-ARC, SciERC NER and SciCite. It fell slightly short of SOTA on JNLPBA, NCBI-disease and GENIA dependency parsing (LAS 90.43 vs. 91.92).
  • Release: code and models are public at github.com/allenai/scibert, and the datasets include the new Paper Field dataset built from the Microsoft Academic Graph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

506 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 493 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1371 · arXiv 1903.10676

@inproceedings{beltagy2019scibert,
  title = {{SciBERT: A Pretrained Language Model for Scientific Text}},
  author = {Beltagy, Iz and Lo, Kyle and Cohan, Arman},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1371},
  eprint = {1903.10676},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1371}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. 2019. Publicly available clinical bert embeddings. In ClinicalNLP workshop at NAACL. find our card
  2. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Z… find
  3. Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In NAACL-HLT, pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics. find
  4. Nigel Collier and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at jnlpba. In NLP-BA/BioNLP. find our card
  5. Tim Dettmers. 2019. TPUs vs GPUs for Transformers (BERT). http: //timdettmers.com/2018/10/17/ tpus-vs-gpus-for-transformers-bert/. Accessed: 2019-02-22. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  7. Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: A resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  8. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. ICLR. find our card
  9. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. In arXiv:1803.07640. find our card
  10. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  11. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv:1904.05342. find our card
  12. Alistair E. W. Johnson, Tom J. Pollard aand Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, , and Roger G. Mark. 2016. find
  13. Mimic-iii, a freely accessible critical care database. In Scientific Data, 3:160035. find our card
  14. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  15. David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Daniel Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. TACL, 06:391–406. find
  16. Nils Reimers and Iryna Gurevych. 2017. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. In EMNLP. find
  17. Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19:i180i182. find
  18. Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (MAS) and applications. In WWW. find
  19. Su Kim, David Martı́nez, Lawrence Cavedon, and Lars Yencken. 2011. Automatic classification of sentences to support evidence based medicine. In BMC Bioinformatics. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR. find
  22. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, We… find
  23. Jens Kringelum, Sonny Kim Kjærulff, Søren Brunak, Ole Lund, Tudor I. Oprea, and Olivier Taboureau. 2016. ChemProt-3.0: a global chemical biology diseases mapping. In Database. find
  24. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. In arXiv:1901.08746. find our card
  25. Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database : the journal of biological databases… find
  26. Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang. 2018. CollaboNet: collaboration of deep neural networks for biomedical named entity recognition. In DTMBio workshop at CIKM. find
  27. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP. find our card
  28. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In arXiv:1902.07669. find
  29. Dat Quoc Nguyen and Karin M. Verspoor. 2019. From pos tagging to dependency parsing for biomedical event extraction. BMC Bioinformatics, 20:1–13. find
  30. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, and Byron C. Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL. find
  31. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT. find