Elamur Research⌕
☰
home / papers / 2026 / Pretraining and Benchmarking Modern Encoders for Latvian

Pretraining and Benchmarking Modern Encoders for Latvian

Paper · 2026 · Arturs Znotins

Summary

Latvian has little presence in the pretraining data of multilingual models, and only a handful of monolingual Latvian encoders exist. This work closes that gap by training a family of Latvian-only encoders on three architectures: RoBERTa, DeBERTaV3, and ModernBERT. The ModernBERT line includes long-context variants that handle up to 8,192 tokens. All models share one training mixture of about 6.43B words, built from web crawls (FineWeb2, HPLT-v2) and curated sources such as LNCC, news, Wikipedia, books, and academic texts. The data was cleaned with deduplication, MinHash LSH near-duplicate removal, and KenLM perplexity filtering. Each model sees 100B tokens, which keeps the comparison fair.

The author also assembles a single Latvian benchmark suite. It covers EuroEval-style diagnostics plus COPA, parsing on the Latvian UD treebank, and a new word sense disambiguation task. The compact lv-deberta-base model scores best overall, ahead of larger multilingual models and earlier Latvian encoders. ModernBERT variants trade a little accuracy for speed and long inputs. All models and evaluation resources are publicly released.

Key points

  • Models and sizes
    • lv-deberta-base has 111M parameters and was trained with RTD at 20% masking, following the CamemBERTaV2 recipe.
    • The ModernBERT-based lv-mbert models come in three sizes: mini (59M), base (136M), and large (377M).
    • lv-roberta-base completes the set.
  • Data and tokenizer
    • The corpus contains about 6.43B words after filtering.
    • A separate 2.5B-token set favoring long documents was used for context extension.
    • Tokenization uses the HPLTv2 Latvian tokenizer with a 32,768-entry vocabulary.
  • Commonsense reasoning (COPA)
    • lv-deberta-base reaches an MCC of 50.9.
    • Most baselines stay below an MCC of 20.
  • Word sense disambiguation
    • lv-deberta-base achieves 78.9% top-1 sense accuracy and 83.6% context–sense match accuracy.
    • It outperforms xlm-roberta-large and all prior Latvian models.
  • UD v2.16 parsing
    • Token-level tagging scores are close to saturation across models.
    • The clearest gains from monolingual pretraining appear in LAS, CLAS, MLAS, and BLEX.
    • ModernBERT models trail lv-deberta-base but bring higher throughput, and their long-context variants help most on extractive QA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LatviaRiga · Latvia5 works · 47 citations🔬UL Institute of Mathematics and Computer ScienceRiga · Latvia2 works · 24 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Arturs Znotins. 2026. Pretraining and Benchmarking Modern Encoders for Latvian. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.40

@inproceedings{znotins2026pretraining,
  title = {{Pretraining and Benchmarking Modern Encoders for Latvian}},
  author = {Arturs Znotins},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.40},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.40}
}

References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Francis Kulumba, Rian Touchent, Éric de la Clergerie, Benoît Sagot, and Djamé Seddah. 2024. Camembert 2.0: A smarter french language model aged to perfection. Preprint, arXiv:2411.08868. find
  2. Wissam Antoun, Benoît Sagot, and Djamé Seddah. 2023. Data-efficient French language modeling with CamemBERTa. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5174– 5185, Toronto, Canada. Association for Computational Linguistics. find
  3. Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X Morris, and Sarath Chandar. 2025. Neobert: A next-generation bert. arXiv preprint arXiv:2502.19587.
  4. Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya M… — An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  6. ‘ Roberts Dar gis. 2022. Corpus of Legal Acts of the Republic of Latvia (Likumi). CLARIN-LV digital library at IMCS, University of Latvia. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025. How to train long-context language models (effectively). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7376– 7399, Vienna, Austria. Association for Computationa…
  9. Normunds Gruzitis, Lauma Pretkalnina, Baiba Saulite, Laura Rituma, Gunta Nespore-Berzkalne, Arturs Znotins, and Peteris Paikens. 2018. Creation of a balanced state-of-the-art multilayer corpus for NLU. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC… find
  10. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.
  11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  12. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I…
  13. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Preprint, arXiv:1907.10529.
  14. ‘ Kristı̄ne Levāne-Petrova, Roberts Dar gis, Kristı̄ne Pokratniece, and Viesturs Jūlijs Lasmanis. 2023. Balanced corpus of modern latvian (LVK2022). CLARIN-LV digital library at IMCS, University of Latvia. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888.
  17. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library.
  18. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. In Proceedings of the 12th Language Resources and Evaluat…
  19. Peteris Paikens, Mikus Grasmanis, Agute Klints, Ilze Lokmane, Lauma Pretkalnin , a, Laura Rituma, Madara Stāde, and Laine Strankale. 2022. Towards Latvian WordNet. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2808–2815, Marseille, France. European Language Re… find
  20. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language. Preprint, arXiv:2506.2…
  21. Matiss Rikters, Rinalds Vı̄ksna, and Edison Marrese-Taylor. 2024. Annotations for exploring food tweets from multiple aspects. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1233–1238, Torino, Italy.… find
  22. Anda Rozukalne, Vineta Kleinberga, and Normunds Grūzı̄tis. 2021. Covid-19 news and audience aggressiveness: Analysis of news content and audience reaction during the state of emergency in latvia (2020–2021). In ENVIRONMENT. TECHNOLOGY. RESOURCES. Proceedings of the International Scientific and Prac… find
  23. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  24. Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2025. Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual NLU tasks. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Lan… find
  25. David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. 2023. Trained on 100 million words and still in shape: BERT meets British National Corpus. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1954–1974, Dubrovnik, Croatia. Association for Computational Lingui… find
  26. Baiba Saulite, Roberts Dargis, and Normunds Gruzitis et al. 2022. Latvian National Corpora Collection – Korpuss.lv. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC). find
  27. Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2025. Seq vs seq: An open suite of paired encoders and decoders. Preprint, arXiv:2507.11412. — [arXiv:2507.11412](https://arxiv.org/abs/2507.11412) find
  28. Arturs Znotins and Guntis Barzdins. 2020. LVBERT: Transformer-Based Model for Latvian Language Understanding. In Human Language Technologies - The Baltic Perspective, volume 328, pages 111–115. IOS Press. find
  29. ‘ Inguna Skadina, Bruno Bakanovs, and Roberts Dar gis. 2025. First steps in benchmarking Latvian in large language models. In Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025), pages 86–95, Tallinn, Estonia. University of… find
  30. Dan Saattrup Smart. 2025. Multiwikiqa: A reading comprehension benchmark in 300+ languages. Preprint, arXiv:2509.04111.
  31. ‘ Uga Spro gis and Matı̄ss Rikters. 2020. What Can We Learn From Almost a Decade of Food Tweets. In In Proceedings of the 9th Conference Human Language Technologies - The Baltic Perspective (Baltic HLT 2020), Kaunas, Lithuania. find
  32. Matej Ulčar and Marko Robnik-Šikonja. 2021. Training dataset and dictionary sizes matter in bert models: the case of baltic languages. In International conference on analysis of images, social networks and texts, pages 162–172. Springer. find
  33. Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438–4450, Online. Association for Computational…
  34. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e…
  35. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/znotins-2026-pretraining-benchmarking-modern-encoders-latvian/