Elamur Research⌕
☰
home / papers / 2024 / BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

Paper · 2024 · Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga et al.

Summary

Frontier systems such as GPT-4 and Med-PaLM 2 perform very well on biomedical NLP. However, they are huge, expensive, reachable only through remote APIs (a problem for patient privacy) and trained on undisclosed data. The authors ask whether a much smaller, domain-focused model can be competitive. They built BioMedLM, a 2.7B-parameter decoder-only Transformer in the GPT-2 style. It uses a tokenizer tailored to biomedical vocabulary and was pre-trained only on PubMed abstracts and full-text articles taken from the Pile. After fine-tuning, it reaches multiple-choice QA accuracy close to that of far larger systems. It clearly beats GPT-Neo 2.7B, a general-English model of the same size and design. It can also write multi-sentence answers to consumer health questions, though it sometimes invents numerical facts. The model is small enough to adapt on one A100 GPU and to run on a laptop, and its training data is fully documented. This makes it an open, inexpensive and privacy-friendly base for in-house biomedical applications. The weights and code are publicly available.

Key points

  • Architecture: 32 layers, hidden size 2560, 20 attention heads, vocabulary of 28,896 tokens, 1024-token context, learned absolute position embeddings. The model was released in December 2022 on the Hugging Face Hub as stanford-crfm/BioMedLM.
  • QA results after fine-tuning: about 57% on MedMCQA (dev) and about 69–70% on MMLU Medical Genetics. The abstract reports 57.3/69.0 and the introduction 57.4/70.0. On BioASQ it scores 95.7, ahead of BioLinkBERT (94.9) and Galactica (94.3).
  • Comparison with GPT-Neo 2.7B: BioMedLM wins on BioASQ, PubMedQA and MedQA, with a 27% accuracy gain on BioASQ. This points to the value of domain-only pre-training.
  • Free-text answers: generated answers to HealthSearchQA questions are mostly accurate, but numerical hallucinations remain. For example, it gives the wrong prevalence for vitamin D deficiency.
  • Later uses by others: state-of-the-art NER on JNLPBA/NCBI-Disease (micro F1 81.31%), the best result on STRING Task 2 for protein interactions, and the highest precision (0.822) for extracting microbiome–disease relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin and Christopher D. Manning. 2024. BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. arXiv:2403.18421.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.18421 · arXiv 2403.18421

@misc{bolton2024biomedlm,
  title = {{BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text}},
  author = {Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin and Christopher D. Manning},
  year = {2024},
  doi = {10.48550/arxiv.2403.18421},
  eprint = {2403.18421},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.18421}
}

References

74 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anmol Arora and Ananya Arora. The promise of large language models in health care. The Lancet, 401(10377):641, 2023. doi: 10.1016/s0140-6736(23)00216-7. — [doi:10.1016/s0140-6736(23)00216-7](https://doi.org/10.1016/s0140-6736%2823%2900216-7) find
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text, 2019. URL https://arxiv.org/abs/1903.10676. ↗
  3. Asma Ben Abacha and Dina Demner-Fushman. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2228–2234, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1215.… find
  4. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this software, please cite it using these metadata. find
  5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive… — GPT-NeoX-20B: An Open-Source Autoregressive Language Model
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  7. Qiuhui Chen, Xinyue Hu, Zirui Wang, and Yi Hong. MedBLIP: Bootstrapping language-image pre-training from 3D medical images and texts, 2023. URL https://arxiv.org/abs/2305.10799. — [arXiv:2305.10799](https://arxiv.org/abs/2305.10799) ↗
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  9. Jennifer Cobbe, Michael Veale, and Jatinder Singh. Understanding accountability in algorithmic supply chains. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, page 1186–1197, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400… — [doi:10.1145/3593013.3594073](https://doi.org/10.1145/3593013.3594073) find
  10. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. URL https://arxiv.org/abs/2205. 14135. ↗
  11. Harm De Vries. Go smol or go home, 2023. model-size-vs-compute-overhead/. find
  12. URL https://www.harmdevries.com/post/Tobias Deußer, Lars Hillebrand, Christian Bauckhage, and Rafet Sifa. Informed named entity recognition decoding for generative language models, 2023. URL https://arxiv.org/abs/2308.07791. — [arXiv:2308.07791](https://arxiv.org/abs/2308.07791) ↗
  13. Joshua C. Feblowitz, Adam Wright, Hardeep Singh, Lipika Samal, and Dean F. Sittig. Summarization of clinical information: A conceptual model. Journal of Biomedical Informatics, 44(4):688–699, 2011. doi: 10.1016/j.jbi.2011.03.008. — [doi:10.1016/j.jbi.2011.03.008](https://doi.org/10.1016/j.jbi.2011.03.008) find
  14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https://arxiv.org/abs/2101. 00027. ↗
  15. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of GPT-3, 2023. URL https://arxiv.org/abs/2209.12356. ↗
  16. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, W… find
  17. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021. doi: 10.1145/ 3458754.
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. ↗
  19. Huggingface. Huggingface/tokenizers: fast state-of-the-art tokenizers optimized for research and production, 2019. URL https://github.com/huggingface/tokenizers. ↗
  20. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, Jul 2021. ISSN 2076-3417. doi: 10.3390/app11146421. URL http://dx.doi.org… ↗
  21. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering, 2019. URL https://arxiv.org/abs/1909. 06146. ↗
  22. Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. GeneGPT: Augmenting large language models with domain tools for improved access to biomedical information, 2023. URL https://arxiv.org/ abs/2304.09667. ↗
  23. Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, Rumman Chowdhury, Alex Engler, Peter Henderson, Yacine Jernite, Seth Lazar, Stefano Maffulli, Alondra Nelson, Joelle Pineau, Aviya Skowron, Dawn… find
  24. Sidd Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Christopher D. Manning Dan Jurafsky, Christopher Potts, Christopher Ré, and Percy Liang. Mistral — a journey towards reproducible language model training, 20… find
  25. Nikitha Karkera, Sathwik Acharya, and Sucheendra K. Palaniappan. Leveraging pre-trained language models for mining microbiome-disease relationships. BMC Bioinformatics, 24(290), 2023. doi: https://doi.org/10.1186/s12859-023-05411-z. — [doi:10.1186/s12859-023-05411-z](https://doi.org/10.1186/s12859-023-05411-z) ↗
  26. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020. URL https://arxiv.org/abs/2004.04906. ↗
  27. Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital H…
  28. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, Sep 2019. doi: 10.1093/bioinformatics/btz682. URL https://doi.org/10.1093%2Fbi…
  29. Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. Summary of ChatGPT/GPT-4 research and perspective towards the future of large language mo… — Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  30. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101. ↗
  31. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Analyzing leakage of personally identifiable information in language models, 2023. URL https://arxiv.org/abs/2302.00539. ↗
  32. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022. doi: 10.1093/bib/bbac409. find
  33. Mason Marks and Claudia E. Haupt. AI chatbots, health privacy, and challenges to HIPAA compliance. JAMA, 330(4):309, 2023. doi: 10.1001/jama.2023.9458. — [doi:10.1001/jama.2023.9458](https://doi.org/10.1001/jama.2023.9458) find
  34. Bertalan Meskó and Eric J. Topol. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. npj Digital Medicine, 6(1), 2023. doi: 10.1038/s41746-023-00873-0. — [doi:10.1038/s41746-023-00873-0](https://doi.org/10.1038/s41746-023-00873-0) find
  35. MosaicML. Composer. https://github.com/mosaicml/composer/, 2021. ↗
  36. Luke Murray, Divya Gopinath, Monica Agrawal, Steven Horng, David Sontag, and David R Karger. MedKnowts: Unified documentation and information retrieval for electronic health records. In The 34th Annual ACM Symposium on User Interface Software and Technology. ACM, October 2021. doi: 10.1145/3472749.3… — [doi:10.1145/3472749.3474814](https://doi.org/10.1145/3472749.3474814) find
  37. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems, 2023a. URL https://arxiv.org/abs/2303.13375. ↗
  38. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models ou… find
  39. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  40. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA : A large-scale multi-subject multi-choice dataset for medical domain question answering, 2022. URL https://arxiv.org/abs/2203.14371. ↗
  41. Gilchan Park, Byung-Jun Yoon, Xihaier Luo, Vanessa López-Marrero, Patrick Johnstone, Shinjae Yoo, and Francis J. Alexander. Comparative performance evaluation of large language models for extracting molecular interactions and pathway knowledge, 2023. URL https://arxiv.org/abs/ 2307.08813. ↗
  42. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Sou… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  43. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training, 2021. URL https://arxiv.org/abs/2104.10350. ↗
  44. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://api.semanticscholar.org/CorpusID: 160025533. ↗
  45. Vin Sachidananda, Jason S. Kessler, and Yi an Lai. Efficient domain adaptation of language models via adaptive tokenization, 2021. URL https://arxiv.org/abs/2109.07460. ↗
  46. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computatio…
  47. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning, 2022. URL https://arxiv.org/abs/2202. 05924. ↗
  48. Nigam H. Shah, David Entwistle, and Michael A. Pfeffer. Creation and adoption of large language models in medicine. JAMA, 330(9):866, 2023. doi: 10.1001/jama.2023.14217. — [doi:10.1001/jama.2023.14217](https://doi.org/10.1001/jama.2023.14217) find
  49. Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training NLP models: A concise overview, 2020. URL https://arxiv.org/abs/2004.08900. — [arXiv:2004.08900](https://arxiv.org/abs/2004.08900) ↗
  50. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023a. doi: 10.1038/s41586-023-06291-2.
  51. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Y… — Towards Expert-Level Medical Question Answering with Large Language Models
  52. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Na… find
  53. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022. URL https://arxiv.org/abs/2211.09085. ↗
  54. Arun James Thirunavukarasu, Darren Shu Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Ting. Large language models in medicine. Nature Medicine, 29(8):1930–1940, 2023. doi: 10.1038/s41591-023-02448-8. — [doi:10.1038/s41591-023-02448-8](https://doi.org/10.1038/s41591-023-02448-8) find
  55. Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C. Comeau, Rezarta Islamaj, Aadit Kapoor, Xin Gao, and Zhiyong Lu. Opportunities and challenges for ChatGPT and large language models in biomedicine and health, 2023. URL https://ar… find
  56. Together. Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models, May 2023a. URL https://www.together.ai/blog/redpajama-models-v1. ↗
  57. Together. RedPajama: An open dataset for training large language models, October 2023b. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023. URL ht…
  59. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, and et al. An overview of the BioASQ large-scale biomedical semantic indexing and question answering… — [doi:10.1186/s12859-015-0564-6](https://doi.org/10.1186/s12859-015-0564-6) find
  60. Tom van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees G. M. Snoek, and Marcel Worring. Open-ended medical visual question answering through prefix tuning of language models, 2023. URL https://arxiv.org/abs/2303.05977. — [arXiv:2303.05977](https://arxiv.org/abs/2303.05977) ↗
  61. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https://arxiv.org/abs/1706. 03762. ↗
  62. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. ↗
  63. Mengqian Wang, Manhua Wang, Fei Yu, Yue Yang, Jennifer Walker, and Javed Mostafa. A systematic review of automatic text summarization for biomedical literature and EHRs. Journal of the American Medical Informatics Association, 28(10):2287–2297, 2021. doi: 10.1093/jamia/ocab143. — [doi:10.1093/jamia/ocab143](https://doi.org/10.1093/jamia/ocab143) find
  64. Shibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud TPUs. https://cloud.google.com/blog/products/ai-machine-learning/ bfloat16-the-secret-to-high-performance-on-cloud-tpus, 2019. ↗
  65. Zuoxi Yang. Biomedical information retrieval incorporating knowledge graph for explainable precision medicine. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 2486, New York, NY, USA, 2020. Association for Computing… — [doi:10.1145/3397271.3401458](https://doi.org/10.1145/3397271.3401458) find
  66. Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining, 2022a. URL https://arxiv.org/abs/2210.09338. ↗
  67. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining language models with document links, 2022b. URL https://arxiv.org/abs/2203.15827. ↗
  68. Hye Sun Yun, Iain J. Marshall, Thomas Trikalinos, and Byron C. Wallace. Appraising the potential uses and harms of LLMs for medical systematic reviews, 2023. URL https://arxiv.org/abs/ 2305.11828. ↗
  69. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023. URL https://arxiv.org/abs/2301.13848. ↗
  70. Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. Learning to summarize radiology findings, 2018. URL https://arxiv.org/abs/1809.04698. ↗
  71. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — A Survey of Large Language Models
  72. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When does pretraining help? Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, 2021. doi: 10.1145/3462757.3466088. — [doi:10.1145/3462757.3466088](https://doi.org/10.1145/3462757.3466088) find
  73. Weipeng Zhou, Majid Afshar, Dmitriy Dligach, Yanjun Gao, and Timothy Miller. Improving the transferability of clinical note section classification models with BERT and large language model ensembles. In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 125– 130, Toronto, Ca… find
  74. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bolton-2024-biomedlm-2-7b-parameter-language/