Elamur Research⌕
☰
home / papers / 2026 / How multilingual are multilingual LLMs? A case study in Northern Sámi-Finnish Translation

How multilingual are multilingual LLMs? A case study in Northern Sámi-Finnish Translation

Paper · 2026 · Jonne Sälevä and Constantine Lignos

Summary

This case study asks whether current multilingual LLMs can translate between Finnish and Northern Sámi. The two languages are related Uralic languages. Finnish is medium-resourced, while Northern Sámi is an Indigenous language with roughly 25,000 speakers and few digital resources.

The authors first tested six models zero-shot: MADLAD-400 (3B/10B), Gemma 3 27B, Aya 101, Aya Expanse and Ministral 3. They compared greedy decoding with sampling. They then applied supervised finetuning to MADLAD-400 and smaller Gemma 3 models. The finetuning data came from Aulamo et al. (2021) in three variants: plain parallel, bidirectional, and parallel plus synthetic. Evaluation used the UiT and YLE test sets with BLEU and chrF2.

Without finetuning, most models are close to unusable. MADLAD-400 is the only clear exception, and it is also the only model that explicitly lists both languages. A few tens of thousands of parallel sentences sharply improve MADLAD-400. The authors report that it becomes competitive with dedicated bilingual systems for Finnish→Sámi and sets a new state of the art for Sámi→Finnish.

The takeaway for low-resource MT is that a small parallel corpus combined with an encoder-decoder model can be enough to get usable translation.

Key points

  • Data: the UiT training set has 25,106 parallel sentences, doubled to 50,212 in the bidirectional set. The synthetic variant (RBMT-all-bt) has 487,862 pairs, where the Finnish side was produced by rule-based MT. Test sets are UiT (2,000 sentences, general domain) and YLE (151 sentences, news).
  • Zero-shot: MADLAD-400-10B-MT performed best with greedy decoding.
    • Finnish⇒Sámi: 9.04 BLEU on UiT and 5.87 on YLE.
    • Sámi⇒Finnish: 24.24 on UiT and 18.04 on YLE.
    • Gemma 3 27B scored about 0 BLEU with greedy decoding but became usable with sampling.
  • MADLAD-400-3B finetuning (Finnish⇒Sámi, UiT):
    • BLEU rose from 6.22 to 23.22 with the parallel data and to 24.19 with the bidirectional data.
    • The parallel+synthetic run, trained for only 1 epoch, did not improve (6.20).
  • Gemma 3 finetuning: the best run was the 12B model trained for 10 epochs.
    • Finnish⇒Sámi: 12.49 BLEU on UiT and 5.93 on YLE.
    • Sámi⇒Finnish: 14.66 on UiT and 6.12 on YLE.
    • These scores still trail finetuned MADLAD, and unlike the zero-shot setting, greedy decoding beat sampling.
  • TranslateGemma (post-hoc, zero-shot): the 4B, 12B and 27B variants all stayed in single-digit BLEU. The best, 27B, reached 4.47 BLEU (UiT) for Finnish⇒Sámi and 8.61 (UiT) for Sámi⇒Finnish, despite having seen both languages during training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Jonne Sälevä and Constantine Lignos. 2026. How multilingual are multilingual LLMs? A case study in Northern Sámi-Finnish Translation. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.42

@inproceedings{saleva2026how,
  title = {{How multilingual are multilingual LLMs? A case study in Northern Sámi-Finnish Translation}},
  author = {Jonne Sälevä and Constantine Lignos},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.42},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.42}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Han, Michael Han, and Unsloth team. 2023. Unsloth. find
  2. Khalid Alnajjar, Mika Hämäläinen, and Jack Rueter. 2024. Leveraging transformer-based models for predicting inflection classes of words in an endangered Sami language. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, pages 41–48, Helsinki, Finland.… find
  3. Katri Hiovain and Antti Suni. 2025. Does multilingual and multi-speaker modeling improve low-resource TTS? Experiments on Sámi languages. In Proceedings of the 13th edition of the Speech Synthesis Workshop, pages 196–201. ISCA - International Speech Communication Association. find
  4. Mikko Aulamo, Sami Virpioja, Yves Scherrer, and Jörg Tiedemann. 2021. Boosting neural machine translation from Finnish to Northern Sámi with rule-based backtranslation. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 351–356, Reykjavik, Iceland (Online). L… find
  5. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Ch… — MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
  6. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  7. Ronny Paul, Himanshu Buckchash, Shantipriya Parida, and Dilip K. Prasad. 2024. Towards a more inclusive AI: Progress and perspectives in large language model training for the Sámi language. Preprint, arXiv:2405.05777. — [arXiv:2405.05777](https://arxiv.org/abs/2405.05777) find
  8. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  9. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  10. Jack Rueter and Mika Hämäläinen. 2020. FST morphology for the endangered Skolt Sami language. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), pages 250–257, Marseille,… find
  11. Jonne Sälevä and Constantine Lignos. 2024. Language model priors and data augmentation strategies for low-resource machine translation: A case study using Finnish to Northern Sámi. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12949–12956, Bangkok, Thailand. Associati… find
  12. David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov, and Stephan Oepen. 2025. Small languages, big models: A study of continual training on languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguisti… find
  13. Maali Tars, Taido Purason, and Andre Tättar. 2022a. Teaching unseen low-resource languages to large translation models. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 375–380, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  14. Maali Tars, Andre Tattar, and Mark Fishel. 2022b. Cross-lingual transfer from large multilingual translation models to unseen under-resourced languages. Baltic Journal of Modern Computing, 10(3):435– 446. find
  15. Linda Wiechetek, Flammie Pirinen, Maja Lisa Kappfjell, Trond Trosterud, Børre Gaup, and Sjur Nørstebø Moshagen. 2024. The ethical question – use of indigenous corpora for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources… find
  16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2019. Huggin…
  17. Lisa Yankovskaya, Maali Tars, Andre Tättar, and Mark Fishel. 2023. Machine translation for low-resource Finno-Ugric languages. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 762–771, Tórshavn, Faroe Islands. University of Tartu Library. find
  18. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya Model: An Instruction Fine…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saleva-2026-how-multilingual-are-multilingual-llms/