Elamur Research⌕
☰
home / papers / 2026 / The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

Paper · 2026 · Henry He, Johann Frei and Raphael Schmitt

Summary

German clinical NLP lacks strong pre-trained encoders. The existing German biomedical models either use older architectures or were trained on small amounts of data. The authors built ChristBERT, a family of RoBERTa-based models that extends the earlier GeistBERT. To compensate for the scarcity of German clinical text, they assembled a 13.5 GB corpus from three kinds of sources: scientific papers, clinical documents and health-related web pages. They enlarged it with machine-translated English material such as MIMIC-IV Notes and PubMed Central.

Using Whole Word Masking, they compared three ways of adapting to the domain:

  • continuing to train a general German model,
  • training from scratch with a general vocabulary,
  • training from scratch with a domain-adapted BPE vocabulary.

The models were assessed with perplexity and on five downstream tasks (three medical NER, two classification). They beat four existing general and medical German models on four of the five tasks. No single strategy won everywhere. Continued pre-training was cheap and strong on some NER sets, while from-scratch training led on classification and on highly specialized clinical text. The work matters because it gives practical guidance on choosing an adaptation strategy and releases open models.

Key points

  • The pre-training corpus is 13.5 GB of German medical text, supplemented by neural machine translation of English sources (MIMIC-IV Notes, PubMed Central).
  • Three ChristBERT variants were trained: continued pre-training, from scratch with a general vocabulary, and from scratch with a domain-specific BPE tokenizer. All use Whole Word Masking on RoBERTa.
  • Evaluation combined intrinsic perplexity with three NER and two text-classification datasets. ChristBERT reached state-of-the-art on 4 of 5 tasks against four baseline German models.
  • The authors flag possible benchmark leakage in two baselines: GeBERTa saw unlabeled CLEF data, and medBERT.de saw GGPONC. Even so, medBERT.de did not gain a visible edge on NER.
  • The models are on Huggingface (huggingface.co/ChristBERT), and the evaluation code is at gitlab.com/christbert/evaluation.

Interesting ideas

  • I1. When in-domain text in the target language is scarce, translating a large foreign-language domain corpus fills the gap, and it also yields a parallel corpus as a free byproduct.
  • I2. Rather than assuming one recipe, the authors trained the same architecture on the same data under several adaptation regimes, and the best choice turned out to depend on the task.
  • I3. Adapting the subword vocabulary to the domain helped mainly with dense specialist terminology, not across the board.
  • I4. The authors audited whether competing models had already seen benchmark text during pre-training, and noted that raw overlap did not automatically raise scores.
  • I5. Pseudonymization masks in de-identified clinical notes are flagged as unnatural artifacts that may skew what a model learns.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Aligned domain text pairs created this way could serve as test material for translation through the meta-world, checking that meaning graphs survive the round trip.
  • I2 (tree): For gparse on a new genre, we could run both options side by side: fine-tune from the general-treebank model, and train fresh on domain silver data. We would then pick per evaluation set rather than by default.
  • I3 (tokens): Adding domain term lists to the tokenizer and morphology lexicon may pay off precisely on jargon-heavy texts, so its effect should be measured on those texts specifically.
  • I4 (learn): Before trusting gains from silver data or LLM-teacher tiers, we should check them for overlap with test sets and report it explicitly, alongside the negative controls.
  • I5 (tokens): Placeholders and redaction tokens in input corpora should be normalized or tagged as special tokens, so they do not distort tagging and lemmas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany62 works · 48,970 citations🎓University of FreiburgFreiburg im Breisgau · Germany6 works · 1,068 citations🎓University of AugsburgAugsburg · Germany8 works · 38,342 citations

Also on the byline: University Medical Center Freiburg.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 works

Cite

Cite as Henry He, Johann Frei and Raphael Schmitt. 2026. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP. arXiv:2606.03250 · BMC Med Inform Decis Mak 26, 296 (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1186/s12911-026-03728-2 · arXiv 2606.03250

@misc{he2026word,
  title = {{The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP}},
  author = {Henry He, Johann Frei and Raphael Schmitt},
  year = {2026},
  doi = {10.1186/s12911-026-03728-2},
  eprint = {2606.03250},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1186/s12911-026-03728-2}
}

References

96 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, Afzal N, et al. Clinical information extraction applications: a literature review. Journal of biomedical informatics. 2018;77:34–49. find
  2. Dalianis H, Hassel M, Velupillai S. The Stockholm EPR Corpus-characteristics and some initial findings. Proceedings of ISHIMR. 2009;p. 243–249. find
  3. Sager N, Lyman M, Bucknall C, Nhan N, Tick LJ. Natural language processing and the representation of clinical data. Journal of the American Medical Informatics Association. 1994;1(2):142–160. ↗
  4. Borst F, Lyman M, Nhan N, Tick L, Sager N, Scherrer J. TEXTINFO: a tool for automatic determination of patient clinical profiles using text analysis. In: Proceedings of the Annual Symposium on Computer Application in Medical Care; 1991. p. 63. find
  5. Friedman C, Johnson SB, Forman B, Starren J. Architectural requirements for a multipurpose natural language processor in the clinical environment. In: Proceedings of the Annual Symposium on Computer Application in Medical Care; 1995. p. 347. find
  6. Zhou B, Yang G, Shi Z, Ma S. Natural language processing for smart healthcare. IEEE Reviews in Biomedical Engineering. 2022;17:4–18. find
  7. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17. Red Hook, NY, USA: Curran Associates Inc.; 2017. p. 6000–6010.
  8. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); 2019. p. 4171–4186.
  9. Liu Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190711692. 2019;.
  10. Arefeva V, Egger R. When BERT Started Traveling: TourBERT—A Natural Language Processing Model for the Travel Industry. Digital. 2022;2(4):546–559. find
  11. Tayefi M, Ngo P, Chomutare T, Dalianis H, Salvi E, Budrionis A, et al. Challenges and opportunities beyond structured data in analysis of electronic health records. Wiley Interdisciplinary Reviews: Computational Statistics. 2021;13(6):e1549. find
  12. Beltagy I, Lo K, Cohan A. SciBERT: A Pretrained Language Model for Scientific Text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); 2019. p. 3615–3620.
  13. Huang K, Altosaar J, Ranganath R. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:190405342. 2019;.
  14. Peng Y, Yan S, Lu Z. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In: Proceedings of the 18th BioNLP Workshop and Shared Task; 2019. p. 58–65.
  15. Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234–1240.
  16. Lentzen M, Madan S, Lage-Rupprecht V, Kühnel L, Fluck J, Jacobs M, et al. Critical assessment of transformer-based AI models for German clinical notes. JAMIA open. 2022;5(4):ooac087. find
  17. Bressem KK, Papaioannou JM, Grundmann P, Borchert F, Adams LC, Liu L, et al. Medbert. de: A comprehensive german bert model for the medical domain. Expert Systems with Applications. 2024;237:121598. find
  18. Scheible-Schmitt R, Frei J. GeistBERT: Breathing Life into German NLP. In: Das SB, Mishra P, Singh A, Muhammad SH, Ekbal A, Das UK, editors. Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models. Varna, Bulgaria: INCOMA Ltd., Shoumen, BULGARIA; 2025. p. 42–50. Available from: https://aclanthology.org/2025.globalnlp-1.6/. ↗
  19. Edunov S, Ott M, Auli M, Grangier D. Understanding Back-Translation at Scale. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing; 2018. p. 489–500. find
  20. Ng N, Yee K, Baevski A, Ott M, Auli M, Edunov S. Facebook FAIR’s WMT19 News Translation Task Submission. In: Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). Association for Computational Linguistics; 2019. . — Facebook FAIR's WMT19 News Translation Task Submission
  21. Costa-jussà MR, Cross J, Çelebi O, Elbayad M, Heafield K, Heffernan K, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:220704672. 2022;. find
  22. Aronson AR, Lang FM. An overview of MetaMap: historical perspective and recent advances. Journal of the American Medical Informatics Association. 2010;17(3):229–236. ↗
  23. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association. 2010;17(5):507–513. find
  24. Friedman C. A broad-coverage natural language processing system. Proceedings of the AMIA Symposium; 2000. p. 270. In: find
  25. Soysal E, Wang J, Jiang M, Wu Y, Pakhomov S, Liu H, et al. CLAMP–a toolkit for efficiently building customized clinical natural language processing pipelines. Journal of the American Medical Informatics Association. 2018;25(3):331–336. find
  26. Uzuner Ö, South BR, Shen S, DuVall SL. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association. 2011;18(5):552–556. find
  27. Henry S, Buchan K, Filannino M, Stubbs A, Uzuner O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association. 2020;27(1):3–12. find
  28. Stubbs A, Filannino M, Soysal E, Henry S, Uzuner Ö. Cohort selection for clinical trials: n2c2 2018 shared task track 1. Journal of the American Medical Informatics Association. 2019;26(11):1163–1171. ↗
  29. Crestani F, Braschler M, Savoy J, Rauber A, Müller H, Losada DE, et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction: 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9–12, 2019, Proceedings. vol. 11696. Springer Nature; 2019. find
  30. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Advances in Neural Information Processing Systems. vol. 26. Curran Associates, Inc.; 2013. .
  31. Peters ME, Neumann M, Zettlemoyer L, Yih Wt. Dissecting Contextual Word Embeddings: Architecture and Representation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing; 2018. p. 1499–1509. ↗
  32. Joulin A, Grave É, Bojanowski P, Mikolov T. Bag of Tricks for Efficient Text Classification. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers; 2017. p. 427–431. find
  33. Martin L, Muller B, Ortiz Suárez PJ, Dupont Y, Romary L, de la Clergerie É, et al. CamemBERT: a Tasty French Language Model. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics; 2020. p. 7203–7219. Available from: https://www.aclweb.org/anthology/2020.acl-main.645. ↗
  34. Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, et al. Unsupervised Cross-lingual Representation Learning at Scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020. p. 8440–8451.
  35. Scheible R, Frei J, Thomczyk F, He H, Tippmann P, Knaus J, et al. GottBERT: a pure German Language Model. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics; 2024. p. 21237–21250. find
  36. Chan B, Schweter S, Möller T. German’s Next Language Model. In: Proceedings of the 28th International Conference on Computational Linguistics; 2020. p. 6788–6796. — German's Next Language Model
  37. Suárez PJO, Sagot B, Romary L. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In: 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache; 2019. . find
  38. De Vries W, van Cranenburgh A, Bisazza A, Caselli T, van Noord G, Nissim M. BERTje: A Dutch BERT model. arXiv preprint arXiv:191209582. 2019;.
  39. White J. PubMed 2.0. Medical reference services quarterly. 2020;39(4):382–387. find
  40. National Library of Medicine.: PMC Open Access Subset. Accessed on 2025-01-12. Available from: https://pmc.ncbi.nlm.nih.gov/tools/openftlist/. ↗
  41. Liu N, Hu Q, Xu H, Xu X, Chen M. Med-BERT: A pretraining framework for medical records named entity recognition. IEEE Transactions on Industrial Informatics. 2021;18(8):5600–5608. find
  42. Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data. 2023;10(1):1.
  43. Akhtyamova L. Named entity recognition in Spanish biomedical literature: Short review and BERT model. In: 2020 26th Conference of Open Innovations Association (FRUCT). IEEE; 2020. p. 1–7. find
  44. Rubel Schneider ET, Andrioli de Souza JV, Knafou J, Oliveira LE, Gumiel YB, de Oliveira LF, et al. BioBERTpt: a Portuguese neural language model for clinical named entity recognition. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop. 19 November 2020; 2020. . find
  45. Copara J, Knafou J, Naderi N, Moro C, Ruch P, Teodoro D. Contextualized French language models for biomedical named entity recognition. In: 6e Conférence Conjointe Journées D’études Sur La Parole (Jep, 33e Edition), Traitement Automatique Des Langues Naturelles (Taln, 27e Edition), Rencontre Des Etudiants Chercheurs En Informatique Pour Le Traitement Automatique Des Langues (Récital, 22e Edition). Atelier Défi Fouille De Textes. ATALA; AFCP; 2020. p. 36–48. find
  46. Starlinger J, Kittner M, Blankenstein O, Leser U. How to improve information extraction from German medical records. It-Information Technology. 2017;59(4):171–179. find
  47. Hellrich J, Matthies F, Faessler E, Hahn U. Sharing models and tools for processing German clinical texts. In: Digital Healthcare Empowering Europeans. IOS Press; 2015. p. 734–738. find
  48. Christina Lohr SB, Hahn U. Sharing Copies of Synthetic Clinical Corpora without Physical Distribution — A Case Study to Get Around IPRs and Privacy Constraints Featuring the German JSYNCC Corpus. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA); 2018. . find
  49. Borchert F, Lohr C, Modersohn L, Witt J, Langer T, Follmann M, et al. GGPONC 2.0-the German clinical guideline corpus for oncology: Curation workflow, annotation policy, baseline NER taggers. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference; 2022. p. 3650–3660. find
  50. Kittner M, Lamping M, Rieke DT, Götze J, Bajwa B, Jelas I, et al. Annotation and initial evaluation of a large annotated German oncological corpus. JAMIA open. 2021;4(2):ooab025. find
  51. Frei J, Kramer F. Annotated dataset creation through large language models for non-english medical NLP. Journal of Biomedical Informatics. 2023;145:104478. find
  52. Organization WH. The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines. vol. 1. World Health Organization; 1992. find
  53. Sänger M, Weber L, Kittner M, Leser U. Classifying German Animal Experiment Summaries with Multi-lingual BERT at CLEF eHealth 2019 Task 1.; 2019. . find
  54. Dada A, Chen A, Peng C, Smith KE, Idrissi-Yaghir A, Seibold CM, et al. On the Impact of Cross-Domain Data on German Language Models. In: The 2023 Conference on Empirical Methods in Natural Language Processing; 2023. . find
  55. Smith R. Peer review: a flawed process at the heart of science and journals. Journal of the royal society of medicine. 2006;99(4):178–182. find
  56. McKinney W, et al. Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference. vol. 445. Austin, TX; 2010. p. 51–56.
  57. Springer Nature.: Springer Nature Developer Portal – APIs for Research Papers. Accessed on 2024-11-10. Available from: https://dev.springernature.com/. ↗
  58. Achakulvisut T, Acuna D, Kording K. Pubmed Parser: A Python Parser for PubMed Open-Access XML Subset and MEDLINE XML Dataset XML Dataset. Journal of Open Source Software. 2020;5(46):1979. https://doi.org/10.21105/ joss.01979. ↗
  59. winstxnhdw.: nllb-api. Accessed on 2025-02-15. Available from: https://github.com/winstxnhdw/nllb-api. ↗
  60. Freie Universität Berlin.: Refubium – Home. Accessed on 2024-11-10. Available from: https://refubium.fu-berlin.de/. ↗
  61. Wikipedia contributors.: Seiten exportieren – Wikipedia. Accessed on 2024-11-12. Available from: https://de.wikipedia.org/wiki/Spezial:Exportieren. ↗
  62. Gojare S, Joshi R, Gaigaware D. Analysis and design of selenium webdriver automation testing framework. Procedia Computer Science. 2015;50:341–346. find
  63. Kurtovic B.: earwig/mwparserfromhell. Original-date: 2012-05-20T18:45:54Z. Available from: https://github.com/earwig/mwparserfromhell. ↗
  64. Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circulation. 2000;101(23):e215–e220. find
  65. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, et al. The llama 3 herd of models. arXiv preprint arXiv:240721783. 2024;.
  66. Deng N, Boeker M, Scheible R. Building a Scalable Health Information Crawler: Leveraging Apache Nutch for the tala-med Search Engine. In: Proceedings of MIE 2025. Glasgow, Scotland: IOPress; 2025. . find
  67. Khare R, Cutting D, Sitaker K, Rifkin A. Nutch: A flexible and scalable open-source web search engine. Oregon State University. 2004;1:32–32. find
  68. Specht L, Scheible R, Boeker M, Farin-Glattacker E, Kampel N, Schmölz M, et al. Evaluating the Acceptance and Usability of an Independent, Noncommercial Search Engine for Medical Information: Cross-Sectional Questionnaire Study and User Behavior Tracking Analysis. JMIR Hum Factors. 2025 Jan;12:e56941. https://doi.org/10.2196/56941. ↗
  69. Zowalla R, Wetter T, Pfeifer D. Crawling the german health web: Exploratory study and graph analysis. Journal of medical Internet research. 2020;22(7):e17853. find
  70. tblock.: 10kGNAD. Accessed on 2025-02-05. Available from: https://tblock.github.io/10kGNAD/. ↗
  71. Schabus D, Skowron M, Trapp M. One million posts: A data set of german online discussions. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval; 2017. p. 1241–1244. find
  72. Tkachenko M, Malyuk M, Holmanyuk A, Liubimov N.: Label Studio: Data Labeling Software. Accessed on 2025-02-05. Available from: https://github.com/heartexlabs/label-studio. ↗
  73. jfilter.: clean-text. Accessed on 2025-02-05. Available from: https://github.com/ jfilter/clean-text. ↗
  74. Schmitt R.: ChristBERT/sciGNAD tcls · Hugging Face. Accessed on 2025-07-31. Available from: https://huggingface.co/ChristBERT/sciGNAD tcls. ↗
  75. He H.: ChristBERT/sciGNAD · Datasets at Hugging Face. Accessed on 2025-07-31. Available from: https://huggingface.co/datasets/ChristBERT/sciGNAD. ↗
  76. Ott M, Edunov S, Baevski A, Fan A, Gross S, Ng N, et al. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations); 2019. p. 48–53.
  77. Bengio Y, Ducharme R, Vincent P, Jauvin C. A neural probabilistic language model. Journal of machine learning research. 2003;3(Feb):1137–1155.
  78. Dieterich C.: CARDIO:DE. 11588/data/AFYQDY. heiDATA. Available from: https://doi.org/10. ↗
  79. German Federal Institute for Risk Assessment.: AnimalTestInfo. Accessed on 2025-04-28. Available from: https://animaltestinfo.de/. ↗
  80. Tjong Kim Sang EF, De Meulder F. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4; 2003. p. 142–147.
  81. Harbecke D, Chen Y, Hennig L, Alt C. Why only Micro-F1? Class Weighting of Measures for Relation Classification. In: Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP; 2022. p. 32–41. find
  82. Fries J, Weber L, Seelam N, Altay G, Datta D, Garda S, et al. BigBIO: A framework for data-centric biomedical natural language processing. Advances in Neural Information Processing Systems. 2022;35:25792–25806. find
  83. Lhoest Q, del Moral AV, Jernite Y, Thakur A, von Platen P, Patil S, et al. Datasets: A Community Library for Natural Language Processing. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations; 2021. p. 175–184.
  84. Nguyen T, Van Nguyen C, Lai VD, Man H, Ngo NT, Dernoncourt F, et al. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:230909400. 2023;.
  85. Tiedemann J. Parallel Data, Tools and Interfaces in OPUS. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12); 2012. p. 2214–2218. find
  86. He P, Liu X, Gao J, Chen W. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In: International Conference on Learning Representations; 2021. .
  87. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations; 2020. p. 38–45.
  88. Microsoft.: Neural Network Intelligence. Accessed on 2025-05-01. Available from: https://github.com/microsoft/nni/. ↗
  89. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. Journal of machine learning research. 2011;12(Oct):2825–2830. find
  90. Gururangan S, Marasović A, Swayamdipta S, Lo K, Beltagy I, Downey D, et al. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020. p. 8342–8360. find
  91. El Boukkouri H, Ferret O, Lavergne T, Zweigenbaum P. Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference; 2022. p. 2626–2633. find
  92. Warner B, Chaffin A, Clavié B, Weller O, Hallström O, Taghadouini S, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:241213663. 2024;.
  93. Beltagy I, Peters ME, Cohan A. Longformer: The long-document transformer. arXiv preprint arXiv:200405150. 2020;.
  94. Xiong Y, Zeng Z, Chakraborty R, Tan M, Fung G, Li Y, et al. Nyströmformer: A nyström-based algorithm for approximating self-attention. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35; 2021. p. 14138–14148. find
  95. Shalumov V, Haskey H.: HeRo: RoBERTa and Longformer Hebrew Language Models. Available from: https://arxiv.org/abs/2304.11077. ↗
  96. Meineke F, Modersohn L, Loeffler M, Boeker M. Announcement of the German medical text corpus project (GeMTeX). In: Caring is Sharing–Exploiting the Value in Data for Health and Innovation. IOS Press; 2023. p. 835–836. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2026-word-way-strategies-domain-specific/