Elamur Research⌕
☰
home / papers / 2026 / What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

Paper · 2026 · Xinlan Yan, Di Wu, Yibin Lei et al.

Summary

Note: the supplied PDF text gives the title as "What Does Neuro Mean to Cardio?", not "Infect". Hospitals focused on one or a few clinical areas need to know how LLMs perform per specialty, but existing open medical QA benchmarks have no specialty labels. The authors build S-MedQA, an English multiple-choice benchmark derived from MedQA and MedMCQA. GPT-3.5 labels each question with a specialty using five separate prompts combined by majority vote, and medical experts check the labels. A conformal-prediction step then assigns additional specialties to cross-disciplinary items. They fine-tune 8 LLMs on single specialties and test them on all the others. They also use clinical terms curated from SNOMED-CT to measure terminology overlap and changes in token probability. Training on a given specialty rarely gives the best results on that same specialty. Clinically relevant terms also become more probable no matter which specialty is used for training. The authors conclude that the gains come mainly from a general-to-medical domain shift rather than from injecting specialty knowledge, which supports the hypothesis of Zhou et al. (2024).

Key points

  • S-MedQA has 24,306 train, 899 validation and 899 test items covering 15 of the 55 EU-recognized specialties. Each included specialty has more than 500 samples. It is released with code at github.com/nlp4health-lab/S-MedQA.
  • Single prompts reached 72.8–80.2% labeling accuracy. With majority voting, accuracy rose to 90.8–97.8%, but coverage fell from 89.1% to 49.2% as the vote threshold increased (3+/4+/5). Krippendorff's alpha among annotators was 83.6%.
  • The multi-label conformal step was calibrated and tested on 300 expert-annotated items (Jaccard 0.82). It achieved set precision 0.69, set recall 0.52 and 24% exact matches, with 1.55 labels per item on average.
  • In one striking result, fine-tuning on neurology data gave the best cardiology performance.
  • Fine-tuning on high-quality non-medical data lowered the probabilities of clinical terms. The data come mainly from US-centric English sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands151 works · 174,506 citations🏥Amsterdam University Medical CentersAmsterdam · Netherlands4 works

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 worksMedicine · 275 worksHealth Informatics · 214 works

Cite

Cite as Xinlan Yan, Di Wu, Yibin Lei, Christof Monz and Iacer Calixto. 2026. What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.390

@inproceedings{yan2026what,
  title = {{What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs}},
  author = {Xinlan Yan, Di Wu, Yibin Lei, Christof Monz and Iacer Calixto},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.390},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.390}
}

References

36 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francisco Castellanos, J Caufield, Lauren Chan, Christopher Chute, Jaime Cruz-Rojo, Noémi Dahan-Oliel, Jon Davids, Maud de Dieuleveult, Vinicius de Souza, Bert de Vries, et al. 2024. Nucleic Acids Research. The human phenotype ontology in 2024: phenotypes around the world., 52(D1). find
  2. ContactDoctor. 2024. Contactdoctor-bio-medical: A high-performance biomedical language model. https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B. ↗
  3. Ronald Cornet and Nicolette de Keizer. 2008. BMC Med Inform Decis Mak. Forty years of SNOMED: a literature review, 8 Suppl 1(Suppl 1):S2. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). In BERT: Pre-training of deep bidirectional transf…
  5. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). In Is GPT-3 a good data annotator?, pages 11173–11195, Toronto, Canada. Association fo…
  6. Peng Fu, Yiming Zhang, Haobo Wang, Weikang Qiu, and Junbo Zhao. 2023. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. In Revisiting the knowledge injection frameworks, pages 10983–10997, Singapore. Association for Computational Linguistics. find
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. arXiv preprint arXiv:2104.08821. Simcse: Simple contrastive learning of sentence embeddings.
  8. Eduardo C Garrido-Merchán, Cristina González-Barthe, and María Coronado Vaca. 2023. arXiv preprint arXiv:2303.13373. Fine-tuning climatebert transformer with climatext for the disclosure analysis of climate-related financial risks. — [arXiv:2303.13373](https://arxiv.org/abs/2303.13373) find
  9. Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al. 2023. Machine Learning for Health (ML4H). In Llms accelerate annotation for medical information extraction, pages 82–100. PMLR. find
  10. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill… find
  11. Andrew F. Hayes and Klaus Krippendorff. 2007. Communication Methods and Measures. Answering the call for a standard reliability measure for coding data, 1(1):77–89. find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Proceedings of the International Conference on Learning Representations (ICLR). Aligning ai with shared human values.
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Proceedings of the International Conference on Learning Representations (ICLR). Measuring massive multitask language understanding.
  14. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. arXiv preprint arXiv:2106.09685. Lora: Low-rank adaptation of large language models.
  15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. arXiv preprint arXiv:2310.06825. Mistral 7b. find
  16. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. Applied Sciences. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 11(14):6421.
  17. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). In PubMedQA: A dataset for biomedical researc…
  18. Yusong Ke, Hongru Lin, Yuting Ruan, Junya Tang, and Li Li. 2025. Mathematics. Correctness coverage evaluation for medical multiple-choice question answering based on the enhanced conformal prediction framework, 13(9):1538. find
  19. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Proceedings of the 23rd Workshop on Biomedical Natural Language Processing. In MedExQA: Medical question answering benchmark with multiple explanations, pages 167–181, Bangkok, Thailand. Association for Computational Linguistics. find
  20. Mikhail V Koroteev. 2021. arXiv preprint arXiv:2103.11943. Bert: a review of applications in natural language processing and understanding. find
  21. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. arXiv preprint arXiv:2402.10373. Biomistral: A collection of open-source pretrained large language models for medical domains.
  22. Nut Limsopatham. 2021. Proceedings of the Natural Legal Language Processing Workshop 2021. In Effectively leveraging BERT for legal document classification, pages 210–216, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  23. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Briefings in bioinformatics. Biogpt: generative pre-trained transformer for biomedical text generation and mining, 23(6):bbac409. find
  24. Elijah Mayfield and Alan W Black. 2020. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. In Should you fine-tune BERT for automated essay scoring?, pages 151–162, Seattle, WA, USA → Online. Association for Computational Linguistics. arXiv:2402.144… find
  25. Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, et al. 2024. arXiv preprint arXiv:2404.00213. Injecting new knowledge into large language models via supervised fine-tuning. — [arXiv:2404.00213](https://arxiv.org/abs/2404.00213) find
  26. Di Wu, Shaomu Tan, Yan Meng, David Stap, and Christof Monz. 2024. Findings of the Association for Computational Linguistics: ACL 2024. In How far can 100 samples go? unlocking zero-shot translation with tiny multi-parallel data, pages 15092–15108, Bangkok, Thailand. Association for Computational Lin… find
  27. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. Proceedings of the 18th BioNLP Workshop and Shared Task. In ScispaCy: Fast and robust models for biomedical natural language processing, pages 319–327, Florence, Italy. Association for Computational Linguistics.
  28. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal… find
  29. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Proceedings of the Conference on Health, Inference, and Learning. In Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering, Proceedings of Machine Learning Researchvolume 174 of , pages 24…
  30. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Nature. Large language models encode clinical knowledge, 620(7972):172–180.
  31. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Nature Medicine. Toward expert-level medical question answering with large language models, 31(3):943–950. find
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. arXiv preprint arXiv:2307.09288. Llama 2: Open foundation and fine-tuned chat models.
  33. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. arXiv preprint Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Proceedings of the AAAI conference on artificial intelligence. In… find
  34. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. arXiv preprint arXiv:2309.03882. On large language models’ selection bias in multi-choice questions. find
  35. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Advances in Neural Information Processing Systems. Lima: Less is more for alignment, 36. find
  36. Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Findings of the Association for Computational Linguistics: EMNLP 2020. In Question answering with long multiple-span answers, pages 3840–3849, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-what-does-infect-mean-cardio/