Elamur Research⌕
☰
home / papers / 2026 / How to Improve LLMs' Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity

How to Improve LLMs' Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity

Paper · 2026 · Xinhe Shi, Qingcheng Zeng, Weihao Xuan et al.

Summary

LLMs perform unevenly across languages. When target-language training data is scarce, practitioners usually pick a related "donor" language for fine-tuning based on linguistic typology. This work questions whether typology matches how models themselves perceive language similarity. The authors propose three similarity measures derived from model behaviour:

  • Feature-based: cosine similarity between per-task accuracy profiles of each language.
  • Transferability-based: accuracy after LoRA fine-tuning on a donor language, divided by accuracy after fine-tuning directly on the target.
  • Integrated: an element-wise product of the two min-max-normalized matrices.

The similarity matrices are computed with monolingual pre-trained models, to avoid confounds from multilingual pre-training. No existing benchmark met their needs, so they built and released M4CQ-Pro. It is a parallel multiple-choice benchmark spanning 31 languages and 135 tasks, with human-reviewed items.

Validation used three small recent models on 20 target languages. All LLM-derived measures picked better fine-tuning languages than typology, and the integrated measure was strongest. Fine-tuning on another language often beat direct target-language fine-tuning, which the authors call a super-additive transfer effect. This matters for low-resource settings, where choosing the donor language well can nearly substitute for target data.

Key points

  • M4CQ-Pro dataset: 31 languages (20+ medium/low-resource), 135 task categories and 61,518 reviewed questions per language, released under CC BY-NC-SA 4.0. The pipeline combined DeepL translation, GPT-4o retranslation and LLM-based correction, with human review of at least 30% of instances per task.

  • Validation models: Llama-3.2-3B, Qwen3-4B-Base and Gemma-3-4B-PT, fine-tuned with LoRA on an 8:2 train/test split. The 20 target languages include Ukrainian (UK), Russian, Polish and Czech.

  • Best-case counts (Table 5):

    StrategyBest results
    Typology6/33 (applicable cases only)
    Feature measure23/60
    Transferability measure18/60
    Integrated measure40/60

  • Super-additive transfer: cross-lingual fine-tuning outperformed direct fine-tuning for nearly all of the 20 target languages across models.

  • Limitations noted by the authors: many low-resource and morphologically complex languages are missing, and 135 tasks may not cover every dimension of similarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations🔬RIKEN Center for Advanced Intelligence ProjectTokyo · Japan15 works · 957 citations🔬RIKENWako · Japan26 works · 526 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xinhe Shi, Qingcheng Zeng, Weihao Xuan and Linchao Zhu. 2026. How to Improve LLMs' Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.691

@inproceedings{shi2026how,
  title = {{How to Improve LLMs' Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity}},
  author = {Xinhe Shi, Qingcheng Zeng, Weihao Xuan and Linchao Zhu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.691},
  url = {https://doi.org/10.18653/v1/2026.acl-long.691}
}

References

71 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shivam Agarwal. 2013. Data mining: Data mining concepts and techniques. In 2013 International Conference on Machine Intelligence and Research Advancement, pages 203–207. find
  2. Ajitesh Bankula and Praney Bankula. 2025. Cross-linguistic transfer in multilingual nlp: The role of language families and morphology. Preprint, arXiv:2505.13908. find
  3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
  4. Paul K. Benedict. 1990. Japanese/Austro-Tai. Karoma, Ann Arbor. find
  5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
  6. R. Bouckaert, P. Lemey, M. Dunn, S. J. Greenhill, A. V. Alekseyenko, A. J. Drummond, R. D. Gray, M. A. Suchard, and Q. D. Atkinson. 2012. Mapping the origins and expansion of the indo-european language family. Science, 337(6097):957–960. find
  7. C. H. Brown, E. W. Holman, S. Wichmann, and V. Velupillai. 2008. Automated classification of the world’s languages: a description of the method and preliminary results. STUF - Language Typology and Universals, 61(4):285–308. find
  8. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. An exploration of hierarchical attention transformers for efficient long document classification. arXiv preprint. find
  9. Charles Condevaux and Sébastien Harispe. 2023. Lsg attention: Extrapolation of pretrained transformers to long sequences. In Advances in Knowledge Discovery and Data Mining, pages 443–454, Cham. Springer Nature Switzerland. find
  10. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022– 6034, Online. Association for Computational Ling…
  11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  13. M. Dunn, S. J. Greenhill, S. C. Levinson, and R. D. Gray. 2011. Evolved structure of language shows lineage-specific trends in word-order universals. Nature, 473(7345):79–82. find
  14. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint. find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  16. Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.07916.
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. CoRR, abs/2009.03300.
  18. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.
  19. Mark Hudson. 1999. Japanese and Austronesian: An Archeological Perspective on the Proposed Linguistic Links, pages 267–279. find
  20. Matt Gardner Johannes Welbl, Nelson F. Liu. 2017. Crowdsourcing multiple choice science questions.
  21. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  22. Takao Kawamoto. 1977. Toward a comparative japanese-austronesian i. Bulletin of Nara University of Education, 26(1). find
  23. Takao Kawamoto. 1978. Minami kara kita Nihongo. Sanseidō, Tōkyō. find
  24. Takao Kawamoto. 1980. Kōdansha, Tōkyō. find
  25. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. ArXiv, abs/2001.04451.
  26. Jungseob Lee, Seongtae Hong, Hyeonseok Moon, and Heuiseok Lim. 2025. Cross-lingual optimization for language transfer in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15100–15119, Vienna, Austria. Asso… find
  27. M. Paul Lewis, Gary F. Simons, and Charles D. Fennig, editors. 2023. Ethnologue: Languages of the World, 26 edition. SIL International. find
  28. Daoyang Li, Haiyan Zhao, Qingcheng Zeng, and Mengnan Du. 2025a. Exploring multilingual probing in large language models: A cross-language analysis. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025), pages 61–70, Vienna, Austria. Association for Comp… find
  29. Yue Li, Zhixue Zhao, and Carolina Scarton. 2025b. It’s all about in-context learning! teaching extremely low-resource languages to LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29544–29559, Suzhou, China. Association for Computational Linguist… find
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  31. April McMahon and Robert McMahon. 2005. Language Classification by Numbers. Oxford University Press. find
  32. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  33. Roy A. Miller. 1967. The Japanese Language. University of Chicago Press, Chicago. find
  34. Roy A. Miller. 1971. Japanese and the Other Altaic Languages. University of Chicago, Chicago. find
  35. Roy A. Miller. 1975. Japanese-altaic lexical evidence and the proto-turkic "zetacism-sigmatism". In Researches in Altaic Languages, pages 157–172. Akadémiai Kiadé, Budapest. find
  36. Roy A. Miller. 1985b. Externalizing internal rules: Lyman’s law in japanese and altaic. Diachronica, 2(2):137–165. find
  37. Shichirō Murayama. 1957. Vergleichende betrachtung der kasus-suffixe im altjapanischen. In Julius von Farkas and Omeljan Pritsak, editors, Studia Altaica: Festschrift für Nikolaus Poppe zum 60 Geburstag, volume 5 of Ural-Altaische Bibliothek, pages 126– 131. Otto Harrassowitz, Wiesbaden. find
  38. Shichirō Murayama. 1962. Nihongo no tingusugo teki yéso. Minzokugaku kenkyū, 26(3). find
  39. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  40. OpenAI. 2024. Mmmlu dataset. https://huggingface.co/datasets/openai/MMMLU. Accessed: 2025-09-14. ↗
  41. M. Pagel, Q. D. Atkinson, A. S. Calude, and A. Meade. 2013. Ultraconserved words point to deep language ancestry across eurasia. Proceedings of the National Academy of Sciences of the United States of America, 110(21):8471–8476. find
  42. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  43. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
  44. G. J. Ramstedt and Pentti Aalto. 1952. Einführung in die altaische sprachwissenschaft. find
  45. Roy A. Miller. 1979. Old korean and altaic. Ural-Altaische Jahrbucher, 51:1–54. find
  46. Enora Rice, Ali Marashian, Hannah Haynie, Katharina von der Wense, and Alexis Palmer. 2025. Untangling the influence of typology, data and model architecture on ranking transfer languages for cross-lingual pos tagging. Preprint, arXiv:2503.19979. find
  47. Roy A. Miller. 1980. Origins of the Japanese Language. University of Washington, Seattle. find
  48. Martine Robbeets. 2005. Is japanese related to korean, tungusic, mongolic and turkic? find
  49. Roy A. Miller. 1983. Japanese evidence for some altaic denominal verb-stem derivational suffixes. Acta Orientalia Hungarica, 36:391–403. find
  50. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. find
  51. Roy A. Miller. 1985a. Altaic connections of the old japanese negatives. Central Asiatic Journal, 29:35– 84. find
  52. Masayoshi Shibatani. 1990. The Languages of Japan. Cambridge University Press, Cambridge. find
  53. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshe… — Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
  54. Aarohi Srivastava, Abhinav Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv,… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  55. Sergei A. Starostin. 1991. Altaiskaia problema i proiskhozhdenie iaponskogo iazyka. Nauka, Moscow. find
  56. John Street and Roy Andrew Miller. 1975–77. Altaic Elements in Old Japanese, volume 1. Madison, Wisconsin. find
  57. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461.
  58. Søren Wichmann, Eric W. Holman, and Cecil H. (eds.) Brown. 2022. The asjp database (version 20). Available at http://asjp.clld.org/. ↗
  59. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang, Huitao Li, Xin Li, Kunyu Yu, Nan Liu, Qingyu Chen, Douglas Teodoro, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, and Irene Li. 2025. Mmlu-prox: A multilingual benchmark for advanced large language…
  60. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  61. John Charles Street and Roy Andrew Miller. 1973. Japanese and the other altaic languages. Language, 49:950. find
  62. Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, Fei Huang, and Jingren Zhou. 2024. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms. Preprint, arXiv:2411.09116. find
  63. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  64. Mikel Zubillaga, Oscar Sainz, Ainara Estarrona, Oier Lopez de Lacalle, and Eneko Agirre. 2024. Event extraction in Basque: Typologically motivated cross-lingual transfer-learning analysis. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and… find
  65. Nicolas Tranter, editor. 2012. The Languages of Japan and Korea, 1st edition. Routledge. find
  66. A Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers. 2019. How does bert answer questions? Proceedings of the 28th ACM International Conference on Information and Knowledge Management - CIKM ’19. find
  67. The format of all tasks in M4CQ-Pro are consistent with that of M4CQ: – idx (Int32): Unique instance identifier within each task. – question (String): Task prompt. find
  68. Alexander Vovin. 1994. Is japanese related to austronesian? Oceanic Linguistics, 33(2):369–390. Accessed 16 Mar. 2025. find
  69. Alexander Vovin. 2005. The end of the altaic controversy. Central Asiatic Journal, 49:71–132. find
  70. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  71. Dataset Format of M4CQ-Pro – choices (List[String]): Options, supporting variable counts. – answer (List[Int32]): Binary vector indicating correct options (1 for correct, 0 for incorrect), with the same length as options. This schema supports flexible option counts and future extension to multi-answ… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-how-improve-llms-performance-specific/