Elamur Research⌕
☰
home / papers / 2026 / MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning

MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning

Paper · 2026 · Kosei Uemura, David Guzmán, Quang Phuoc Nguyen et al.

Summary

Open LLMs reason well in English but perform poorly on maths and inference tasks in many low-resource languages. Earlier encoder-plus-LLM methods such as LangBridge and MindMerger narrowed this gap only for mid- and high-resource languages. MERLIN attaches a frozen multilingual encoder to a frozen decoder LLM through a small two-layer connector, trained in two stages. In Stage I, the connector follows a curriculum that moves from general to specific data: generic bitext, then translated question pairs, then translated question–answer pairs. In Stage II, the connector is frozen and only DoRA adapters inside the decoder are tuned on English task data. At inference, the model answers target-language questions in English without having seen task data in those languages.

On AfriMGSM, MERLIN beats MindMerger by 12.9 accuracy points and GPT-4o-mini by 15.2. It also gives smaller gains on MGSM, MSVAMP and AfriXNLI. This shows that cheap, parameter-efficient model stacking can substantially improve reasoning in under-served languages.

Key points

  • Two backbone pairings were tested: Metamath 7B with mT5-XL, and Gemma 2 9B with NLLB-600M-distilled. Only the connector and the DoRA weights are trained.
  • With Gemma 2 9B and NLLB, average accuracy versus MindMerger is 76.2 vs 73.4 on MGSM and 79.2 vs 78.3 on MSVAMP. The abstract reports these two gains (+0.9 / +2.8) in the opposite order from the table. On AfriXNLI the gain is +1.6.
  • Baselines include MindMerger, LangBridge, LayAlign, SLAM and QAlign-LoRA. The authors report gains of more than 11 points over all of them on AfriMGSM.
  • Under an equal budget of 12k pairs per language, MERLIN still leads MindMerger, at 78.3 vs 73.4 on MGSM and 80.4 vs 78.3 on MSVAMP. The choice of encoder strongly affects results for low-resource languages.
  • Limitations:
    • All training data is unfiltered machine translation, which causes notable errors for Oromo.
    • Evaluation covers only maths and NLI.
    • A separate model is trained for each benchmark.
    • Code is available at github.com/LLMforLRL/MERLIN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🔬Canadian Institute for Advanced ResearchToronto · Canada28 works · 16,725 citations🎓Ontario Tech UniversityOshawa · Canada8 works · 1,554 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Kosei Uemura, David Guzmán, Quang Phuoc Nguyen, Jesujoba Oluwadara Alabi, En-Shiun Annie Lee and David Ifeoluwa Adelani. 2026. MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.277

@inproceedings{uemura2026merlin,
  title = {{MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning}},
  author = {Kosei Uemura, David Guzmán, Quang Phuoc Nguyen, Jesujoba Oluwadara Alabi, En-Shiun Annie Lee and David Ifeoluwa Adelani},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.277},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.277}
}

References

61 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Are… — MasakhaNER: Named Entity Recognition for African Languages
  2. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024a. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the E…
  3. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, and et al. 2024b. Irokobench: A new benchmark for african languages in the age of large language models. Preprint, arXiv:2406.03368.
  4. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic…
  5. Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. Tower: An open multilingual large language model for translation-related tas…
  6. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, and 1 others. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032.
  7. Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multilinguality with cross-lingual expert language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages…
  8. Happy Buzaaba, Alexander Wettig, David Ifeoluwa Adelani, and Christiane Fellbaum. 2025. Lugha-llama: Adapting large language models for african languages. arXiv preprint arXiv:2504.06536. — [arXiv:2504.06536](https://arxiv.org/abs/2504.06536) find
  9. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. Preprint, arXiv:2310.20246. find
  10. Zoltan Csaki, Pian Pawakapan, Urmish Thakker, and Qiantong Xu. 2023. Efficiently adapting pretrained language models to new languages. Preprint, arXiv:2311.05741.
  11. Tejas Deshpande, Nidhi Kowtal, and Raviraj Joshi. 2024. Chain-of-translation prompting (cotr): A novel prompting technique for low resource languages. Preprint, arXiv:2409.04512. find
  12. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.
  13. Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Wenhu Chen, and Ge Zhang. 2024. Chinese tiny llm: Pretraining a chinese-centric large language model. Preprint, arXiv:2404.04167.
  14. Yuchun Fan, Yongyu Mu, Yilin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, and Jingbo Zhu. 2025. Slam: Towards efficient multilingual reasoning via selective language alignment. Preprint, arXiv:2501.03681.
  15. Poulami Ghosh, Shikhar Vashishth, Raj Dabre, and Pushpak Bhattacharyya. 2024. A morphology-based investigation of positional encodings. Preprint, arXiv:2404.04530. find
  16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  17. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. Preprint, arXiv:2403.14608.
  18. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  19. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365– 1239…
  20. Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, and Fei Yuan. 2024. Mindmerger: Efficient boosting llm reasoning in non-english languages. Preprint, arXiv:2405.17386. — [arXiv:2405.17386](https://arxiv.org/abs/2405.17386) find
  21. Sheriff Issaka, Zhaoyi Zhang, Mihir Heda, Keyi Wang, Yinka Ajibola, Ryan DeMar, and Xuefeng Du. 2024. The ghanaian nlp landscape: A first look. Preprint, arXiv:2405.06818. — [arXiv:2405.06818](https://arxiv.org/abs/2405.06818) find
  22. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, and et al. 2024. Mixtral of experts. Preprint, arXiv:2401.04088. find
  23. Muzammil Khan, Kifayat Ullah, Yasser Alharbi, Ali Alferaidi, Talal Saad Alharbi, Kusum Yadav, Naif Alsharabi, and Aakash Ahmad. 2023. Understanding the research challenges in low-resource language and linking bilingual news articles in multilingual news archive. Applied Sciences, 13(15). find
  24. En-Shiun Annie Lee, Sarubi Thillainathan, Shravan Nayak, Surangika Ranathunga, David Ifeoluwa Adelani, Ruisi Su, and Arya D. McCarthy. 2022. Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation? In Findings of the Association for Computational Linguistic… find
  25. Danni Liu and Jan Niehues. 2025. Middle-layer representation alignment for cross-lingual transfer in fine-tuned llms. Preprint, arXiv:2502.14830.
  26. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. Preprint, arXiv:2402.09353. find
  27. Joanito Agili Lopo and Radius Tanone. 2024. Constructing and expanding low-resource and underrepresented parallel datasets for indonesian local languages. Preprint, arXiv:2404.01009. find
  28. Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024. LLaMAX: Scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10748–10772, Miami, Florida, USA. Association for Comput… find
  29. Stefano Lusito, Edoardo Ferrante, and Jean Maillard. 2023. Text normalization for low-resource languages: the case of ligurian. Preprint, arXiv:2206.07861. find
  30. Xuetao Ma, Wenbin Jiang, and Hua Huang. 2025. Problem-solving logic guided curriculum in-context learning for LLMs complex reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8394–8412, Vienna, Austria. Association for Computational Linguistics. find
  31. Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. Preprint, arXiv:2006.07264. — [arXiv:2006.07264](https://arxiv.org/abs/2006.07264) find
  32. Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Said Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahudeen Abdullahi, Anuoluwapo Aremu, Alipio Jeorge, and Pavel Brazdil. 2022. Naijasenti: A nigerian twitter sentiment cor…
  33. Raymond Ng, Thanh Ngan Nguyen, Yuli Huang, Ngee Chia Tai, Wai Yi Leong, Wei Qi Leong, Xianbin Yong, Jian Gang Ngui, Yosephine Susanto, Nicholas Cheng, and 1 others. 2025. Sea-lion: Southeast asian languages in one network. arXiv preprint arXiv:2504.05747.
  34. Antoine Nzeyimana. 2024. Low-resource neural machine translation with morphological modeling. Preprint, arXiv:2404.02392. find
  35. Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Esther Adenuga, David Ifeoluwa Adelani, and Jimmy Lin. 2025. Improving multilingual math reasoning for african languages. arXiv preprint arXiv:2505.19848. find
  36. Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David Ifeoluwa Adelani. 2024. How good are large language models on african languages? Preprint, arXiv:2311.07978. — [arXiv:2311.07978](https://arxiv.org/abs/2311.07978) find
  37. Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, and Jimmy Lin. 2023. Better quality pre-training data and t5 models for African languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 158–… find
  38. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  39. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695–2709, Singapore. Association for C…
  40. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. Preprint, arXiv:2404.04925. find
  41. Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural Machine Translation for Low-resource Languages: A Survey. ACM Comput. Surv., 55(11):229:1–229:37. find
  42. Zhiwen Ruan, Yixia Li, He Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang, Yun Chen, and Guanhua Chen. 2025. Layalign: Enhancing multilingual reasoning in large language models via layer-wise adaptive fusion and alignment strategy. Preprint, arXiv:2502.11405. find
  43. Fabian David Schmidt, Philipp Borchert, Ivan Vulić, and Goran Glavaš. 2024. Self-distillation for model stacking unlocks cross-lingual NLU in 200+ languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6724–6743, Miami, Florida, USA. Association for Computational… find
  44. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. Preprint, arXiv:2210.03057.
  45. Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum Learning: A Survey. Int. J. Comput. Vision, 130(6):1526–1565. find
  46. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, and et al. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  47. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, and et al. 2024b. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. find
  48. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, and et al. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. find
  49. Atnafu Tonja, Fazlourrahman Balouchzahi, Sabur Butt, Olga Kolesnikova, Hector Ceballos, Alexander Gelbukh, and Thamar Solorio. 2024. NLP progress in indigenous Latin American languages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisti… find
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  51. Kosei Uemura, Mahe Chen, Alex Pejovic, Chika Madu-abuchi, Yifei Sun, and En-Shiun Annie Lee. 2024. AfriInstruct: Instruction tuning of African languages for diverse tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13571–13585, Miami, Florida, USA. Association fo… find
  52. Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2023. Scaling laws for BERT in low-resource settings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7771–7789, Toronto, Canada. Association for Computational Linguistics. find
  53. Yudong Wang, Chang Ma, Qingxiu Dong, Lingpeng Kong, and Jingjing Xu. 2023. A challenging benchmark for low-resource learning. Preprint, arXiv:2303.03840. find
  54. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Preprint, arXiv:1704.05426.
  55. Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024. Adapting large language models for document-level machine translation. Preprint, arXiv:2401.06468. — [arXiv:2401.06468](https://arxiv.org/abs/2401.06468) find
  56. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2024. Lexc-gen: Generating data for extremely low-resource languages with large language models and bilingual lexicons. Preprint, arXiv:2402.14086. find
  57. Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. Lang-Bridge: Multilingual reasoning without multilingual supervision. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7502–7522, Ban… find
  58. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  59. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. Preprint, arXiv:2309.12284.
  60. Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. Question translation training for better multilingual reasoning. Preprint, arXiv:2401.07817. find
  61. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, and et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. Preprint, arXiv:2402.07827.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uemura-2026-merlin-multi-stage-curriculum-alignment/