Elamur Research⌕
☰
home / papers / 2026 / KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts

KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts

Paper · 2026 · Christian Autenried and Cosimo Persia

Summary

General-purpose Norwegian encoders know little about the language of hospital records, which is full of abbreviations, informal phrasing and specialist vocabulary. The authors address this with KliniskVestBERT, three encoders built by continuing the pretraining of Nb-BERT-large, NorBERT3-large and ModernBERT on real patient documentation from the Helse Vest hospitals. The corpus contains about 16.2 million documents from 2019, 2021 and 2022, in both bokmål and nynorsk. Before training, every document went through an automatic de-identification tool developed for the project. It swaps personal details for realistic surrogates and shifts dates consistently within each document. The resulting models were tested on three synthetic Norwegian medical benchmarks and two in-house applied tasks. Each clinical model scored better than the general model it started from, and the clinical models also needed far fewer fine-tuning epochs. The work shows that domain-adaptive pretraining on representative clinical data pays off for Norwegian medical NLP, even when strong general models already exist.

Key points

  • Corpus: 16,211,880 documents drawn from 715 of the more than 1,100 document types in the DIPS health record system. It holds roughly 5.1 billion tokens and 456 million sentences, with a median document length of about 203 tokens. The split is 89% bokmål and 11% nynorsk. The year 2020 was left out to avoid a Covid-19 bias.
  • De-identification covers names, locations, care units, dates, ID numbers, phone numbers and emails. Each item is replaced by a plausible surrogate, applied consistently within a document. All dates in a document move by the same random offset of 365–1,825 days.
  • For the ModernBERT variant, the masking rate starts at 15% for the first 5% of training. It then rises gradually to 30%, mirroring how the original ModernBERT was trained.
  • F1 results for Kl-Nb-BERT against Nb-BERT-large: MedMCQA 39.17 vs 28.44, MedMQP 89.64 vs 84.12 and Synergi 94.58 vs 88.02.
  • Kl-ModernBERT beat ModernBERT on Nor-DeID (90.49 vs 88.8) and on AISMEC stroke-call classification (82.14 vs 79.52).
  • On a medical NER task, the clinical models reached their best checkpoint after 1–2 epochs, while the baselines needed about 8–10.

Interesting ideas

  • I1. When anonymising, keep surrogate names consistent within a document and shift all its dates by one offset, so references and timelines stay intact. This keeps the cleaned text usable for tasks that depend on document-level structure.
  • I2. When continuing pretraining, reproduce the base model's original training regime, here a masking curriculum ramping from 15% to 30%, instead of imposing a new one. This lowers the risk of disturbing what the model already learned.
  • I3. Leave out a data period known to be atypical (the Covid year) rather than letting it skew the distribution, even at the cost of volume.
  • I4. Treat the number of fine-tuning epochs needed to reach the best dev score as a signal of how useful the prior adaptation was, alongside final accuracy.
  • I5. Compare models using one best checkpoint each, with no checkpoint averaging, so that differences reflect the training data rather than post-hoc tricks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Apply consistent renaming and date shifting to fables and test texts, then check that the built world of the text stays isomorphic. This gives a cheap invariance test for entity tracking and event ordering.
  • I2 (tree): When adapting gparse to a new treebank or UD dialect, warm-start with the schedule and hyperparameters from its original training and only shift them gradually.
  • I3 (learn): Flag silver data from unusual sources or genres and hold it out of training. It can still serve as a separate stress set for evaluation.
  • I4 (learn): Log epochs-to-best for each training run, so it is visible whether new silver data or teacher tiers actually give the model a head start.
  • I5 (learn): Use one fixed selection rule for every model variant when reporting gains, so improvements can be attributed cleanly to the change being tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

Cite

Cite as Christian Autenried and Cosimo Persia. 2026. KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts. arXiv:2606.01904.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.01904 · arXiv 2606.01904

@misc{autenried2026kliniskvestbert,
  title = {{KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts}},
  author = {Christian Autenried and Cosimo Persia},
  year = {2026},
  doi = {10.48550/arxiv.2606.01904},
  eprint = {2606.01904},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.01904}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guttorm Brattebø. Artificial Intelligence Support in Medical Emergency Calls – AISMEC project. 2025. url: https://www.helse-bergen.no/avdelinger/kirurgisk-serviceklinikk/fag-og-forskingsavdelinga/aismec/ (visited on 07/01/2025). ↗
  2. Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: CoRR abs/1810.04805 (2018). arXiv: 1810.04805. url: http://arxiv.org/abs/1810.04805. ↗
  3. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. 2020. arXiv: 1904.05342 [cs.CL]. url: https://arxiv.org/abs/1904.05342. ↗
  4. Mandar Joshi et al. “SpanBERT: Improving Pre-training by Representing and Predicting Spans”. In: Transactions of the Association for Computational Linguistics 8 (Jan. 2020), pp. 64–77. issn: 2307-387X. doi: 10.1162/tacl_a_00300. eprint: https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00300/1923170/tacl\_a\_00300.pdf. url: https://doi.org/10.1162/tacl%5C_a%5C_00300. ↗
  5. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 2017. arXiv: 1412.6980 [cs.LG]. url: https://arxiv.org/abs/1412.6980. ↗
  6. Per Kummervold, Freddy Wetjen, and Javier de la Rosa. “The Norwegian Colossal Corpus: A Text Corpus for Training Large Norwegian Language Models”. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference. Ed. by Nicoletta Calzolari et al. Marseille, France: European Language Resources Association, June 2022, pp. 3852–3860. url: https : //aclanthology.org/2022.lrec-1.410/. ↗
  7. Yinhan Liu et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019. arXiv: 1907.11692 [cs.CL]. url: https://arxiv.org/abs/1907.11692. ↗
  8. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. 2019. arXiv: 1711. 05101 [cs.LG]. url: https://arxiv.org/abs/1711.05101. ↗
  9. Jørgen Aarmo Lund et al. “Instruction-guided deidentification with synthetic test cases for Norwegian clinical text”. In: Proceedings of the 5th Northern Lights Deep Learning Conference (NLDL). Ed. by Tetiana Lutchyn, Adı́n Ramı́rez Rivera, and Benjamin Ricaud. Vol. 233. Proceedings of Machine Learning Research. PMLR, Sept. 2024, pp. 145–152. url: https://proceedings.mlr.press/v233/lund24a.html. ↗
  10. Clara H. McCreery et al. Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs. 2020. arXiv: 2008.13546 [cs.IR]. ↗
  11. P. D. Ngo et al. “Domain-Specific Pretraining and Evaluation of NorDeClin-BERT for ICD-10 Code Prediction in Norwegian Clinical Texts”. In: JMIR AI 66153 (2025). Forthcoming. doi: 10.2196/66153. url: https://preprints.jmir.org/preprint/66153. ↗
  12. Casey Lynnette Overby, Peter Tarczy-Hornoch, and Dina Demner-Fushman. “The potential for automated question answering in the context of genomic medicine: an assessment of existing resources and properties of answers”. en. In: BMC Bioinformatics 10.Suppl 9 (Sept. 2009), S8. ↗
  13. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”. In: Proceedings of the Conference on Health, Inference, and Learning. Ed. by Gerardo Flores et al. Vol. 174. Proceedings of Machine Learning Research. PMLR, Apr. 2022, pp. 248–260. url: https://proceedings.mlr.press/v174/pal22a.html. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering ↗
  14. Adam Paszke et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. 2019. arXiv: 1912.01703 [cs.LG]. url: https://arxiv.org/abs/1912.01703. ↗
  15. Amit Persad, Eleni Stroulia, and Sarah Forgie. “A novel approach to virtual patient simulation using natural language processing”. In: Medical Education 50.11 (2016), pp. 1162– 1163. doi: https://doi.org/10.1111/medu.13197. eprint: https://asmepublications. onlinelibrary.wiley.com/doi/pdf/10.1111/medu.13197. url: https://asmepublications. onlinelibrary.wiley.com/doi/abs/10.1111/medu.13197. ↗
  16. Alec Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision. 2022. arXiv: 2212.04356 [eess.AS]. url: https://arxiv.org/abs/2212.04356. ↗
  17. David Samuel et al. NorBench – A Benchmark for Norwegian Language Models. 2023. arXiv: 2305.03880 [cs.CL]. url: https://arxiv.org/abs/2305.03880. ↗
  18. Imre Solti et al. “Automated classification of radiology reports for acute lung injury: Comparison of keyword and machine learning based natural language processing approaches”. In: 2009 IEEE International Conference on Bioinformatics and Biomedicine Workshop. Washington, DC: IEEE, Nov. 2009. find
  19. Benjamin Warner et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. 2024. arXiv: 2412.13663 [cs.CL]. url: https://arxiv.org/abs/2412.13663. ↗
  20. Jannicke Slettli Wathne. KI for uønskede legemiddelhendelser. 2025. url: https://www.helse-bergen.no (visited on 07/01/2025). ↗
  21. Thomas Wolf et al. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. 2020. arXiv: 1910.03771 [cs.CL]. url: https://arxiv.org/abs/1910.03771. ↗
  22. Mitchell Wortsman et al. Stable and low-precision training for large-scale vision-language models. 2023. arXiv: 2304.13013 [cs.LG]. url: https://arxiv.org/abs/2304.13013. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/autenried-2026-kliniskvestbert-bert-model-specialised-norwegian/