Elamur Research⌕
☰
home / papers / 2021 / Can Language Models be Biomedical Knowledge Bases?

Can Language Models be Biomedical Knowledge Bases?

Paper · 2021 · Mujeen Sung, Jinhyuk Lee, Sean Yi et al.

Summary

This work asks whether pre-trained language models can act as knowledge bases for expert biomedical facts, rather than the general trivia tested by LAMA. To study this, the authors built BioLAMA, a probing benchmark of about 49K subject–relation–object triples drawn from CTD, UMLS and Wikidata. The triples cover 36 relations, and domain experts helped write a manual prompt for each one. The benchmark is harder than earlier probes in two ways. Objects are often several subwords long, and triples whose answer appears inside the subject string were removed.

The authors probed BERT, BioBERT and Bio-LM with two kinds of prompts, manual prompts and OptiPrompt. Multi-token answers were produced with a confidence-based decoding scheme. As a non-LM reference, they used the BEST entity search tool.

The biomedical models beat the baseline, but the best scores remain low. Further analysis shows that predictions mostly depend on the prompt template and barely on the subject entity. The authors conclude that current LMs are not yet usable as biomedical KBs, and BioLAMA gives future work a challenging benchmark.

Key points

  • Data sources: BioLAMA has roughly 22K triples with 15 relations from CTD, 21.2K with 16 relations from UMLS (2020AB), and 5.8K with 5 relations from Wikidata. Each relation is split 40:10:50 into train, dev and test sets.
  • Answer length: Objects average 4.52 subwords, compared with 1.00 in LAMA. Entities are capped at 10 subwords, which keeps about 90% of them.
  • Best result: Bio-LM with OptiPrompt reaches 7.28% Acc@1 and 18.51% Acc@5 on macro average. The BEST IE baseline scores 5.21% and 11.56%.
  • Prompt bias: All LMs show a prompt bias above 0.6, measured as the Pearson correlation between predictions with and without the subject. The IE system's bias is noticeably lower.
  • Synonym variance: LMs vary less than the IE system when subjects are swapped for synonyms. However, BERT combines the lowest variance with the highest bias, so this stability likely reflects bias rather than real awareness of synonyms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations🏢NaverSeongnam-si · South Korea15 works · 11,528 citations🎓Korea UniversitySeoul · South Korea8 works · 9,146 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksBiochemistry, Genetics and Molecular Biology · 116 worksMolecular Biology · 114 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksKnowledge graphs & ontologies · 103 works

Cite

Cite as Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim and Jaewoo Kang. 2021. Can Language Models be Biomedical Knowledge Bases?. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.388

@inproceedings{sung2021can,
  title = {{Can Language Models be Biomedical Knowledge Bases?}},
  author = {Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim and Jaewoo Kang},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.388},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.388}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620,… find
  2. Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267– D270. find
  3. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
  4. Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2020. Comparative Toxicogenomics Database (CTD): update 2021. Nucleic Acids Research, 49(D1):D1138–D1143. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943– 5959, Online. As… find
  7. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  8. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
  9. Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. Best: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PloS one, 11(10):e0164680. find
  10. Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146–157, Online. Association for Computation…
  11. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  12. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  13. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818, Online. Association for Computational Linguistics. find
  14. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
  15. Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3641–3650, Online. Association for Computational Linguistics. find
  16. Houcemeddine Turki, Thomas Shafee, Mohamed Ali Hadj Taieb, Mohamed Ben Aouicha, Denny Vrandečić, Diptanshu Das, and Helmi Hamdi. 2019. Wikidata: A large-scale collaborative ontological medical database. Journal of biomedical informatics, 99:103292. find
  17. Andra Waagmeester, Gregory Stupp, Sebastian Burgstaller-Muehlbacher, Benjamin M Good, Malachi Griffith, Obi L Griffith, Kristina Hanspers, Henning Hermjakob, Toby S Hudson, Kevin Hy-biske, et al. 2020. Science forum: Wikidata as a knowledge graph for the life sciences. Elife, 9:e52614. find
  18. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Co…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sung-2021-can-language-models-be-biomedical/