Elamur Research⌕
☰
home / papers / 2026 / Where do LLMs currently stand on biomedical NER in both clean and noisy settings ?

Where do LLMs currently stand on biomedical NER in both clean and noisy settings ?

Paper · 2026 · Christophe Ye and Cassie S. Mitchell

Summary

This paper revisits how well current large language models handle biomedical named entity recognition, a task long dominated by domain-adapted BERT encoders. The authors benchmark five recent LLMs (Llama-3.1-8B, Phi-4, Gemma3-12B, Mistral-3.1-24B, GPT-4o) against BioBERT, PubMedBERT, BiolinkBERT and ModernBERT on five corpora: NCBI-Disease, BC5CDR, MedMentions-ST21PV, MT-Samples and VAERS. LLMs are tested with in-context learning, using faiss-based retrieval of similar training examples and an inline delimiter tagging scheme. The best open model, Mistral-3.1, is also fine-tuned. Further experiments cover limited training data, noisy annotations, and an error analysis that sorts mistakes into five categories. Domain-specific BERT models, especially BiolinkBERT and PubMedBERT, remain the most accurate, and they are far cheaper to train and run. Still, newer LLMs have narrowed the gap considerably. They also outperform BERT when annotated data is scarce or labels are heavily corrupted. This points to practical roles such as cold-start data generation or selective use on specific data slices.

Key points

  • Across datasets, the newer Mistral-3.1 (March 2025) improves on Llama-3.1 (July 2024) by roughly 10–60% in relative terms. More in-context examples (k = 1, 3, 5, 10) help only the stronger models, Mistral-3.1 and GPT-4o.
  • Smaller LLMs often break the required output format, and their annotation tends to be overly conservative. GPT-4o and Mistral, by contrast, tend to over-annotate, producing invalid entity tags.
  • Mistral-3.1 surpasses the BERT models when fewer than about 60 annotated abstracts are available. With 50–60% noisy training labels, LLMs also beat BERT trained on equally noisy data.
  • Compute cost differs sharply on MM-ST21PV. Fine-tuning Mistral-3.1 required 22 hours on a single H100, whereas BioLinkBERT finished training in 50 minutes using an A40. Generating predictions took the LLM 9 hours, compared with less than an hour for BERT.
  • Code is released at github.com/CY/biomedicalNER.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksNamed entity recognition · 214 works

Cite

Cite as Christophe Ye and Cassie S. Mitchell. 2026. Where do LLMs currently stand on biomedical NER in both clean and noisy settings ?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.51

@inproceedings{ye2026where,
  title = {{Where do LLMs currently stand on biomedical NER in both clean and noisy settings ?}},
  author = {Christophe Ye and Cassie S. Mitchell},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.51},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.51}
}

References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Sali… — Phi-4 Technical Report
  2. Saadullah Amin, Katherine Ann Dunfield, Anna Vechkaeva, and Guenter Neumann. 2020. A data-driven approach for noise reduction in distantly supervised biomedical relation extraction. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 187–194, Online. Association fo… find
  3. Ted T Ashburn and Karl B Thor. 2004. Drug repositioning: identifying and developing new uses for existing drugs. Nature reviews Drug discovery, 3(8):673– 683. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. Necva Bölücü, Maciej Rybinski, Xiang Dai, and Stephen Wan. 2024. An adaptive approach to noisy annotations in scientific information extraction. Information Processing Management, 61(6):103857. find
  6. Sudha Cheerkoot-Jalim and Kavi Kumar Khedo. 2021. Literature-based discovery approaches for evidence-based healthcare: a systematic review. Health and Technology, 11:1205 – 1217. find
  7. Priya Desai, Natalie Telis, Benjamin Lehmann, Keith Bettinger, Kizito Jonathan, and Somalee Datta. 2018. Scireader*: A cloud-based recommender system for biomedical literature. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. pages 4171–4186.
  9. Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: A resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  10. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. Preprint, arXiv:2301.00234.
  11. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The faiss library. Preprint, arXiv:2401.08281.
  12. Jason A. Fries, Troy Mayfield, Kenneth Brimacombe, Michael M. Bronstein, David Silver, David Wehner, et al. 2022. Bigbio: A large-scale biomedical corpus. Journal of Biomedical Informatics, 135:104037. find
  13. Gemma. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, and etc... 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  15. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1–23.
  16. Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about gpt-3 in-context learning for biomedical ie? think again. Preprint, arXiv:2203.08410.
  17. Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. 2024. Algorithmic progress in language models. Preprint, arXiv:2403.05812. — [arXiv:2403.05812](https://arxiv.org/abs/2403.05812) find
  18. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  19. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. 2024. Improving large language models for clinical named entity recognition via prompt engineering. Preprint, arXiv:2303.16416. — [arXiv:2303.16416](https://arxiv.org/abs/2303.16416) find
  20. Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, 171:108189. find
  21. David Kartchner, Selvi Ramalingam, Irfan Al-Hussaini, Olivia Kronick, and Cassie Mitchell. 2023. Zero-shot information extraction for clinical meta-analysis using large language models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 396–405, Toronto, Ca… find
  22. David Kartchner, Haydn Turner, Christophe Ye, Irfan Al-Hussaini, Batuhan Nursal, Albert J. B. Lee, Jennifer Deng, Courtney Curtis, Hannah Cho, Eva L. Du-varis, Coral Jackson, Catherine E. Shanks, Sarah Y. Tan, Selvi Ramalingam, and Cassie S. Mitchell. 2025. Trialsieve: A comprehensive biomedical inf… find
  23. Vipina K Keloth, Yan Hu, Qianqian Xie, Xueqing Peng, Yan Wang, Andrew Zheng, Melih Selek, Kalpana Raja, Chih Hsuan Wei, Qiao Jin, Zhiyong Lu, Qingyu Chen, and Hua Xu. 2024. Advancing entity recognition in biomedicine via instruction tuning of large language models. Bioinformatics, 40(4):btae163. find
  24. Anna Kirkpatrick, Chidozie Onyeze, David Kartchner, Stephen Allegri, Davi Nakajima An, Kevin McCoy, Evie Davalbhakta, and Cassie S. Mitchell. 2022. Optimizations for computing relatedness in biomedical heterogeneous information networks: Semnet 2.0. Big Data and Cognitive Computing, 6(1). find
  25. Z. Kusynová, G.M. Pauletti, H.A van den Ham, H.G.M. Leufkens, and A.K. Mantel-Teeuwisse. 2022. Unmet medical need as a driver for pharmaceutical sciences – a survey among scientists. Journal of Pharmaceutical Sciences, 111(5):1318–1324. find
  26. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  27. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
  28. Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068. find
  29. Mistral. 2025. Mistral. Mistral website to mistral-small-3-1 model. find
  30. Sunil Mohan, Rico Angell, Nicholas Monath, and Andrew McCallum. 2021. Low resource recognition and linking of biomedical concepts from a large ontology. In Proceedings of the 12th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, BCB ’21, page 1–10. ACM. find
  31. Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls concepts. Preprint, arXiv:1902.09476. — [arXiv:1902.09476](https://arxiv.org/abs/1902.09476) find
  32. Milad Moradi, Kathrin Blagec, and Matthias Samwald. 2021. Deep learning models are not robust against noise in clinical text. Preprint, arXiv:2108.12242. — [arXiv:2108.12242](https://arxiv.org/abs/2108.12242) find
  33. Monica Munnangi, Sergey Feldman, Byron C Wallace, Silvio Amir, Tom Hope, and Aakanksha Naik. 2024. On-the-fly definition augmentation of llms for biomedical ner. Preprint, arXiv:2404.00152. find
  34. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the 56th Annual Meeting of the Associatio…
  35. Aurélie Névéol, Hercules Dalianis, Sumithra Velupillai, Guergana Savova, and Pierre Zweigenbaum. 2018. Clinical natural language processing in languages other than english: Opportunities and challenges. Journal of biomedical semantics, 9:12. find
  36. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex… — GPT-4o System Card
  37. Jacqueline Peng, Mengge Zhao, James Havrilla, Cong Liu, Chunhua Weng, Whitney Guthrie, Robert Schultz, Kai Wang, and Yunyun Zhou. 2020. Natural language processing (nlp) tools in extracting biomedical concepts from research articles: a case study on autism spectrum disorder. BMC Medical Informatics… find
  38. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  39. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. Preprint, arXiv:2109.01652.
  40. Yishu Wei, Yu Deng, Cong Sun, Mingquan Lin, Hongmei Jiang, and Yifan Peng. 2024. Deep learning with noisy labels in medical prediction problems: a scoping review. Preprint, arXiv:2403.13111. — [arXiv:2403.13111](https://arxiv.org/abs/2403.13111) find
  41. Olivier J Wouters, Martin McKee, and Jeroen Luyten. 2020. Estimated research and development investment needed to bring a new medicine to market, 2009-2018. Jama, 323(9):844–853. find
  42. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. Preprint, arXiv:2203.15827.
  43. PubMed. 2025. Pubmed. Website of pubmed. find
  44. John Schulman and Thinking Machines Lab. 2025. Lora without regret. Thinking Machines Lab: Connectionism. Https://thinkingmachines.ai/blog/lora/. find
  45. Sonish Sivarajkumar and Yanshan Wang. 2022. Healthprompt: A zero-shot learning paradigm for clinical natural language processing. Preprint, arXiv:2203.05061. — [arXiv:2203.05061](https://arxiv.org/abs/2203.05061) find
  46. S Tian, Q Jin, L Yeganova, et al. 2023. Opportunities and challenges for chatgpt and large language models in biomedicine and health. arxiv. arXiv preprint arXiv:2306.10070. find
  47. Hugo Touvron, Louis Martin, Kevin Stone, and Peter Albert etc... 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  48. Marceli Wac, Raul Santos-Rodriguez, Chris McWilliams, and Christopher Bourdeaux. 2024. Capturing requirements for a data annotation tool for intensive care: Experimental user-centered design study. Preprint, arXiv:2309.16500. — [arXiv:2309.16500](https://arxiv.org/abs/2309.16500) find
  49. Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-where-do-llms-currently-stand/