Elamur Research⌕
☰
home / papers / 2026 / Out-Of-Tune rather than Fine-Tuned: How Pre-training, Fine-tuning and Tokenization Affect Semantic Similarity in a Historical, Non-Standardized Domain

Out-Of-Tune rather than Fine-Tuned: How Pre-training, Fine-tuning and Tokenization Affect Semantic Similarity in a Historical, Non-Standardized Domain

Paper · 2026 · Stella Verkijk and Piek Vossen

Summary

The paper asks whether encoder language models capture the meaning of domain-specific concepts in Early Modern Dutch from the Dutch East India Company (VOC) archives (1602–1799). This setting is hard for two reasons: spelling was not yet standardized, and the texts carry extra noise from Handwritten Text Recognition. To select strings representing domain concepts such as commodities, the authors drew on a dataset in which experts had annotated named entities and semantic roles. They then compared multilingual, contemporary-Dutch and domain-specific models, both before and after fine-tuning. Their measure is a new interpretable metric, a semantic Signal-to-Noise Ratio (SNR), which shows how far similarity within a concept rises above the model's similarity between random strings. The domain-pretrained GloBERTise performed best. General multilingual and monolingual models did about equally poorly, and larger models or more pre-training data did not help. Most strikingly, fine-tuning usually made the concept representations semantically worse, which hints at catastrophic forgetting and reliance on shortcuts. The authors conclude that domain world knowledge counts for more than tokenization, a result relevant to many low-resource settings.

Key points

  • SNR metric: SNR is expressed in standard deviations above random-pair similarity. For example, GloBERTise (base, last layer) scores 1.25σ for commodity strings (n=435) against random strings (n=1000).
  • Model comparison: RobBERT and BERTje have similar subtoken counts and overlap, yet their results differ. The authors read this as a sign that domain distributional semantics matter more than the tokenizer.
  • Spelling-variant case study: The test pairs included olie/olij, oliphanten/eliphanten and contanten/comptanten. BERTje entirely missed the similarity between the two spellings of "oil". RobBERT linked unrelated words that share the subtoken "ol".
  • Shared success: All models clustered enslaved patients and enslaving agents well, possibly because these mentions have low lexical variety.
  • Ethics and scope: The authors warn that a faithful domain model inherits colonial biases, such as treating people as commodities. Decoder models were not tested. The work is part of the GLOBALISE project (NWO), and the code and data are public.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Vrije Universiteit AmsterdamAmsterdam · Netherlands42 works · 3,372 citations🔬Huygens Institute for History and Culture of the NetherlandsAmsterdam · Netherlands3 works · 12 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksTokenization & segmentation · 119 works

Cite

Cite as Stella Verkijk and Piek Vossen. 2026. Out-Of-Tune rather than Fine-Tuned: How Pre-training, Fine-tuning and Tokenization Affect Semantic Similarity in a Historical, Non-Standardized Domain. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.45

@inproceedings{verkijk2026out,
  title = {{Out-Of-Tune rather than Fine-Tuned: How Pre-training, Fine-tuning and Tokenization Affect Semantic Similarity in a Historical, Non-Standardized Domain}},
  author = {Stella Verkijk and Piek Vossen},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.45},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.45}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sophie Arnoult, Brecht Nijman, and Leon van Wissen. 2025. Fine-grained named-entity recognition for the east-india company domain. Anthology of Computers and the Humanities, 3:953–967. ↗
  2. Sophie I Arnoult, Lodewijk Petram, and Piek Vossen. 2021. Batavia asked for advice. pretrained language models for named entity recognition in historical texts. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 21–30. find
  3. Wietse De Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model. arXiv preprint arXiv:1912.09582. ↗
  4. Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim. 2020. What’s so special about bert’s layers? a closer look at the nlp pipeline in monolingual and multilingual models. Findings of the Association for Computational Linguistics: EMNLP. find
  5. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. Robbert: a dutch roberta-based language model. Findings of the Association for Computational Linguistics: EMNLP.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  7. Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. 2021. Towards interpreting and mitigating shortcut learning behavior of nlu models. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ↗
  8. Leon Eshuijs, Shihan Wang, and Antske Fokkens. 2025. Short-circuiting shortcuts: Mechanistic investigation of shortcuts in text classification. arXiv preprint arXiv:2505.06032. ↗
  9. Ali Forooghi, Shaghayegh Sadeghi, and Jianguo Lu. 2024. Whitening not recommended for classification tasks in llms. In The 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), page 285. find
  10. Nathan Godey, Éric Clergerie, and Benoît Sagot. 2024. Anisotropy is inherent to self-attention in transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35–48. find
  11. Thomas Goerttler and Klaus Obermayer. 2022. Similarity of pre-trained and fine-tuned representations. arXiv preprint arXiv:2207.09225. ↗
  12. Ganesh Jawahar, Benoıt Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  14. Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 5312–5327. ↗
  15. Stella Verkijk, Pia Sommerauer, and Piek Vossen. 2024. Studying language variation considering the reusability of modern theories, tools and resources for annotating explicit and implicit events in centuries old text. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 174–187. find
  16. Stella Verkijk, Piek Vossen, and Pia Sommerauer. 2025. Language models lack temporal generalization and bigger is not better. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20629– 20637. find
  17. Enrique Manjavacas and Lauren Fonteyn. 2022. Non-parametric word sense disambiguation for historical languages. In Proceedings of the 2nd international workshop on natural language processing for digital humanities, pages 123–134. find
  18. Ivan Vulic, Edoardo M Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pre-trained language models for lexical semantics. Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  19. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
  20. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the english comparative correlative. Proceedings of the Conference on Empirical Methods in Natural Language Processing.
  21. Marius Mosbach. 2023. Analyzing pre-trained and fine-tuned language models. In The Big Picture Workshop, page 123. find
  22. Sina Bagheri Nezhad, Ameeta Agrawal, and Rhitabrat Pokharel. 2025. Beyond data quantity: Key factors driving performance in multilingual language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 225– 239. ↗
  23. Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. 2020. Boil: Towards representation change for few-shot learning. arXiv preprint arXiv:2008.08882. ↗
  24. Ruixiang Tang, Dehan Kong, Longtao Huang, and 1 others. 2023. Large language models can be lazy learners: Analyze shortcuts in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4645–4657. find
  25. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. ↗
  26. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual bert? Proceedings of the 5th Workshop on Representation Learning for NLP, page 120.
  27. Yichu Zhou and Vivek Srikumar. 2022. A closer look at how fine-tuning changes bert. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1046–1061. find
  28. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. ↗
  29. Wondimagegnhue Tufa, Lisa Beinborn, and Piek Vossen. 2023. A wordnet view on crosslingual contextualized language models. University of the Basque Country in Donostia-San Sebastian Basque Country, page 14. find
  30. Betty Van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. 2019. How does bert answer questions? a layer-wise analysis of transformer representations. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1823–1832. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/verkijk-2026-out-tune-rather-than-fine/