Elamur Research⌕
☰
home / papers / 2026 / Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation

Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation

Paper · 2026 · Rian Touchent, Nathan Godey and Éric Villemonte de la Clergerie

Summary

Scientific articles in PubMed Central are internally uneven. Clinical case descriptions and pedagogical review passages appear alongside methods boilerplate and low-information text. Filtering whole documents, as FineWeb-Edu does for web pages, therefore throws away useful material. The authors instead label PMC Open Access text one paragraph at a time, tagging each for document type, domain, educational quality and language. They use a two-stage setup. Llama-3.1-70B annotates a sample of 400K paragraphs, and a fine-tuned XLM-RoBERTa classifier then extends those labels to roughly 133M paragraphs. The resulting Biomed-Enriched corpus makes it possible to compose training mixes by filtering on quality and upsampling clinical content. It also yields about 2M clinical case paragraphs that are freely accessible, unlike privacy-restricted hospital notes. Continual pretraining with these mixes improves both decoders and encoders. Curated decoder mixes converge about three times faster. An encoder trained only on public data equals BioClinical-ModernBERT. This suggests that careful data selection can partly substitute for access to restricted clinical corpora.

Key points

  • Heterogeneity within articles: 91.6% of sampled PMC articles mix several document types, and educational scores differ by 2.6 points on average inside a single article (1–5 scale).
  • Two-stage pipeline: Llama-3.1-70B labeled 400K paragraphs (0.33% of the corpus), and XLM-RoBERTa then labeled about 133M paragraphs. The corpus, the classifier and the best encoder are released.
  • Decoder results: OLMo2-7B-stage1 was continually pretrained on 33.6B additional tokens.
    • Clinical upsampling added about 4 points on MMLU ProfMed.
    • Educational filtering added about 1 point on MedQA and MedMCQA.
    • The combined BE-All mix hit the baseline's performance using about one third of the tokens.
  • Encoder results: The best recipe scored 77.3% average F1 on 11 tasks, versus 77.1% for BioClinical-ModernBERT. It used 2.5× fewer tokens (2T + 67B vs. 2T + 169B) and only public PubMed/PMC data.
  • Decay-phase effects: Each paragraph type helped different tasks. Study paragraphs improved NER, Review paragraphs improved GAD relation extraction, and Clinical Case paragraphs improved SocialHistory.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France84 works · 25,414 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Rian Touchent, Nathan Godey and Éric Villemonte de la Clergerie. 2026. Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1713

@inproceedings{touchent2026biomed,
  title = {{Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation}},
  author = {Rian Touchent, Nathan Godey and Éric Villemonte {de la Clergerie}},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1713},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1713}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Johan Högberg, Ulla Stenius, and Anna Korhonen. 2016. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics, 32(3):432–440. find
  2. Àlex Bravo, Janet Piñero, Núria Queralt-Rosinach, Michael Rautschka, and Laura I Furlong. 2015. Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC Bioinformatics, 16:55. find
  3. Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. 2024. Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats. arXi… — [arXiv:2405.19538](https://arxiv.org/abs/2405.19538) find
  4. Zeming Chen, Alejandro Hernández-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne H… — MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  6. Nigel Collier and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73–78. COL-ING.
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  8. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: A resource for disease name recognition and concept normalization. Journal of Biomedical Informatics, 47:1–10. find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR).
  10. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  11. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577. Association for Computational Lingu…
  12. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-Wei H Lehman, Leo A Celi, and Roger G Mark. 2023. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10:1.
  13. Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-Wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific Data, 3:160035.
  14. Prescott Klassen, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen. 2014. Annotating clinical events in text snippets for phenotype detection. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2753–2757, Reykjavik, Iceland. European Language Res… find
  15. Martin Krallinger, Obdulia Rabal, Analia Lourenco, and 1 others. 2017. Overview of the biocreative vi chemical-protein interaction track. In Proceedings of the BioCreative VI Workshop, pages 141–146. find
  16. Yanis Labrak, Adrien Bazoge, Richard Dufour, Beatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. 2022. FrenchMedMCQA: A French multiple-choice question answering dataset for medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Informa… find
  17. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, Bang…
  18. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, and 1 others. 2024. Datacomp-lm: In search of the next generation of training sets for language models. In Advances in Neural Information Processing Systems, volume 37.
  19. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068. find
  20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2381–2391, Brussels, Belgium. Associ… find
  21. Edward T. Moseley, Joy T. Wu, Jonathan Welt, John Foote, Patrick D. Tyler, David W. Grant, Eric T. Carlson, Sebastian Gehrmann, Franck Dernoncourt, and Leo Anthony Celi. 2020. A corpus for detecting high-context medical conditions in intensive care patient notes focusing on frequently readmitted pat… find
  22. National Library of Medicine. 2024. PMC open access subset. https://www.ncbi.nlm.nih.gov/ pmc/tools/textmining/. Accessed 2025-03-29. ↗
  23. Ishna Neamatullah, Margaret Douglass, Li-Wei Lehman, Andrew Reisner, Mauricio Villarroel, William J Long, Peter Szolovits, George B Moody, Roger G Mark, and Gari D Clifford. 2008. Automated de-identification of free-text medical records. BMC Medical Informatics and Decision Making, 8:32. find
  24. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025. 2 ol… find
  25. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24…
  26. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557.
  27. Sampo Pyysalo and Sophia Ananiadou. 2014. Anatomical entity mention recognition at literature scale. Bioinformatics, 30(6):868–875. find
  28. Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. 2025. Bioclinical modernbert: A state-of-the-art long-context encoder for biomedical and clinical nlp. Preprint, arXiv:250… — [arXiv:2506.10896](https://arxiv.org/abs/2506.10896) find
  29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  30. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  31. Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. 2025. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341. find
  32. Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. 2024. PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31(9):1833–1843. find
  33. Meliha Yetisgen and Lucy Vanderwende. 2017. Automatic identification of substance abuse from social history in clinical text. In Artificial Intelligence in Medicine: 16th Conference on Artificial Intelligence in Medicine, AIME 2017, pages 171–181. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/touchent-2026-biomed-enriched-data-efficient-biomedical/