Elamur Research⌕
☰
home / papers / 2026 / Are the LLMs Capable of Maintaining at Least the Language Genus?

Are the LLMs Capable of Maintaining at Least the Language Genus?

Summary

Multilingual LLMs often reply in a language other than the prompt's, and their accuracy varies widely across languages. Earlier work mostly treated this as simple adherence versus deviation. This paper asks whether genealogical relatedness shapes the behaviour. The authors reanalyse MultiQ (Holtermann et al., 2024), replacing its coarse family-level grouping (English / same family / other) with WALS genera. They pose two questions:

  • When a model abandons the prompt language, does it stay within the same genus?
  • Does a correct answer in one language predict a correct answer in a related language of that genus?

Besides the six original open-weight models, they add the fully open Apertus-8B and generate its answers with the original MultiQ pipeline.

Genus fidelity splits the models into three clear tiers: Llama-2 lowest, Mistral/Mixtral/Qwen in the middle and Apertus highest. Genus-level effects do appear, but the amount of training data per language explains much more of the behaviour, and model families follow distinct multilingual strategies. For lower-resource languages, linguistic proximity therefore offers limited protection against data imbalance.

Key points

  • Data and genus mapping: MultiQ has 27,400 parallel questions (200 per language, 137 languages).

    • Languages were mapped to genera through WALS (2,662 entries) using WALS and ISO 639-3 codes.
    • The result is 47 genera in 21 families. Eighteen genera contain two or more languages (101 languages in total), led by Indic (16) and Germanic, Bantu and Slavic (11 each).

  • Models and tools:

    • Models: Llama-2-Chat in 7B, 13B and 70B, Mistral-7B-Instruct-v0.1, Mixtral-8x7B-Instruct-v0.1, Qwen1.5-7B-Chat and Apertus-8B. The paper's table labels the middle Llama size "14b", while the appendix uses "13b". Apertus covers about 1,800 languages, with roughly 40% non-English data.
    • Answers were judged by gpt-4-0125-preview, and response languages were identified with GlotLID.

  • Metric: FidelityScore is the share of prompts whose response genus equals the prompt's genus.

  • FidelityScore results:

    ModelScore
    Llama-2-7B17.3
    Llama-2-13B23.1
    Llama-2-70B27.9
    Qwen1.5-7B-Chat70.5
    Mixtral-8x7B73.9
    Mistral-7B75.0
    Apertus-8B92.3

    Apertus falls to roughly 0.5 or lower only for Tano and Quechuan. High fidelity does not guarantee correct answers.

  • Code: available at github.com/IDSIA-NLP/GenusPref.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/38cn6xjcqa4p

@inproceedings{mitrovic2026are,
  title = {{Are the LLMs Capable of Maintaining at Least the Language Genus?}},
  author = {Mitrović, Sandra and Kletz, David and Dolamic, Ljiljana and Rinaldi, Fabio},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/38cn6xjcqa4p},
  url = {https://doi.org/10.63317/38cn6xjcqa4p}
}

References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mina Almasi and Ross Kristensen-McLachlan. 2025. Alignment drift in CEFR-prompted LLMs for interactive Spanish tutoring. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 70–88, Vienna, Austria. Association for Computational Linguist… find
  2. Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards.… find
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… find our card
  4. Yiyi Chen, Qiongxiu Li, Russa Biswas, and Johannes Bjerva. 2025. Large language models are easily confused: A quantitative metric, security implications and typological analysis. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3810–3827, Albuquerque, New Mexico. Assoc… find
  5. Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching large language models new languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–2… find
  6. Matthew S. Dryer. 1989. Large linguistic areas and language sampling. Studies in Language, 13:257–292. find
  7. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online (v2020.4). Zenodo. find
  8. Vansh Gupta, Sankalan Pal Chowdhury, Vilém Zouhar, Donya Rooein, and Mrinmaya Sachan. 2025. Multilingual performance biases of large language models in education. find
  9. Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. 2025. DOVE: A large-scale multi-dimensional predictions dataset towards meaningful LLM evaluation. In Findings of the Association for Computational Linguistics: ACL 2025,… find
  10. Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank $urech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabol%ec, Yixuan Xu, Mic… find
  11. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the elementary multilingual capabilities of large language models with MultiQ. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4476–4494, Bangkok, Thailand. Association for Computational Li… find
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  14. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. GlotLID: Language identification for low-resource languages. In The 2023 Conference on Empirical Methods in Natural Language Processing. find our card
  15. Fengyuan Liu, Nouar AlDahoul, Gregory Eady, Yasir Zaki, and Talal Rahwan. 2025a. Self-reflection makes large language models safer, less biased, and ideologically neutral. find
  16. Wei Liu, Sony Trenous, Leonardo F. R. Ribeiro, Bill Byrne, and Felix Hieber. 2025b. Xrag: Cross-lingual retrieval-augmented generation. find
  17. Colin P Masica. 1993. The indo-aryan languages. Cambridge University Press. find
  18. Krenare Pireva Nuci, Paul Landes, and Barbara Di Eugenio. 2024. RoBERTa low resource fine tuning for sentiment analysis in Albanian. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14146–14151, Torino… find
  19. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Je! Belgum, Irwan Bello, Jak… find our card
  20. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. Multilingual instruction tuning with just a pinch of multilinguality. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2304–2317, Bangkok, Thailand. Association for Computation… find
  21. Michael C Shapiro. 1989. A primer of modern standard Hindi. Motilal Banarsidass Publ. find
  22. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  23. Khanh-Tung Tran, Barry O’Sullivan, and Hoang D. Nguyen. 2025. Scaling test-time compute for low-resource languages: Multilingual reasoning in llms. find
  24. Dawid Wi"niewski, Antoni Solarski, and Artur Nowakowski. 2025. Exploring the feasibility of multilingual grammatical error correction with a single LLM up to 9B parameters: A comparative study of 17 models. In Proceedings of Machine Translation Summit XX: Volume 1, pages 231– 247, Geneva, Switzerlan… find
  25. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. find our card
  26. Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. Beyond english-centric llms: What language do multilingual language models think in? find
  27. Han Zhou, Xingchen Wan, Ivan Vuli’, and Anna Korhonen. 2023. Survival of the most influential prompts: E\#cient black-box prompt search via clustering and pruning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13064–13077, Singapore. Association for Computational Lin… find
  28. Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. find
  29. Question translation training for better multilingual reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8411–8423, Bangkok, Thailand. Association for Computational Linguistics. 7. find
  30. Language Resource References Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Pro… find our card
  31. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  32. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the elementary multilingual capabilities of large language models with MultiQ. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4476–4494, Bangkok, Thailand. Association for Computational Li… find
  33. A. Statistics The languages in MultiQ span 47 genera, 18 of which are represented by multiple languages. Table 4 shows the distribution of these 18 genera. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mitrovic-2026-are-llms-capable-maintaining-least/