Group-Merger: A LoRA-based Framework for Multilingual Continual Learning
Paper · 2026 · Weijian yi, Hongliang Li and Jinan Xu
Summary
The paper addresses multilingual continual learning (MCL). In this setting, a language model receives new language-specific tasks one after another. It must learn them without forgetting earlier ones and without replaying old data. Assigning a separate LoRA module to each task prevents forgetting, but the number of parameters grows with every task and the modules cannot learn from each other. Merging all modules into one avoids the growth but causes interference that worsens as tasks accumulate.
Group-Merger takes a middle path. Incoming tasks are placed into groups with a fixed capacity n. Once a group is full, its LoRA modules are merged with SVD into a single module. To limit interference, training adds an orthogonality penalty only among tasks in the same group. The strength of that penalty is scaled by how similar the tasks are, using sentence-embedding similarity, so that related languages can share knowledge. Modules from earlier groups also take part in the forward pass. On the XNLI and MLQA benchmarks, the framework outperforms existing LoRA-based continual learning baselines while storing fewer extra modules.
Key points
- Storage stays bounded: the number of extra modules at inference equals ⌈T/n⌉ for T tasks and group size n. No historical data needs to be replayed.
- Average performance improves over prior state-of-the-art by 1.75% on XNLI. The gain on MLQA is only 0.02%.
- In the Llama-3B ablation, removing previous-module incorporation lowers XNLI AP from 74.23 to 63.68 and raises forgetting from 1.51 to 11.82. Task-aware orthogonality outperforms plain orthogonality.
- Group size is a trade-off between accuracy and storage. As the number of extra modules goes from 1 to 2 to 3, XNLI AP rises from 72.25 to 74.23 to 76.52.
- The paper's own merging method outperforms Weight Average, Task Arithmetic and Ties-Merging. For example, it reaches 77.05 on XNLI at n=2, compared with 68.59 for Ties-Merging.
Institutions
ChinaFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works
🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works
Cite
Cite as Weijian yi, Hongliang Li and Jinan Xu. 2026. Group-Merger: A LoRA-based Framework for Multilingual Continual Learning. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).
@inproceedings{yi2026group,
title = {{Group-Merger: A LoRA-based Framework for Multilingual Continual Learning}},
author = {Weijian yi, Hongliang Li and Jinan Xu},
booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
year = {2026},
doi = {10.18653/v1/2026.mellm-1.30},
url = {https://doi.org/10.18653/v1/2026.mellm-1.30}
}References
- Qwen3-1.7B Llama3-3B Llama3-8B Qwen3-8B 73.66 74.23 79.35 79.43 60.31 62.71 64.28 63.41 find
- Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marcaurelio Ranzato. 2019. On tiny episodic memories in continual learning. arxiv. Learning, 6(7). *Table 5: The results of the XNLI and MLQA benchmarks with different architectures an… find
- Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Ghotkar. 2023. Task arithmetic with lora for continual learning. arXiv preprint arXiv:2311.02428. \#\# Conclusion In this paper, we introduce a novel framework Group-Merger that addresses catastrophic forgetting in incrementally learning new langua… — [arXiv:2311.02428](https://arxiv.org/abs/2311.02428) find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/yi-2026-group-merger-lora-based-framework/