Elamur Research⌕
☰
home / papers / 2026 / Sparse Adapter Fusion for Continual Learning in NLP

Sparse Adapter Fusion for Continual Learning in NLP

Paper · 2026 · Min Zeng, Xi Chen, Haiqin Yang et al.

Summary

Continual learning systems must take on a sequence of new tasks without losing what they learned on earlier ones, a failure known as catastrophic forgetting. Adapter-based methods such as AdapterCL and ACM protect old tasks by attaching extra modules, but their size grows with every task, and related tasks share little of what they learn. The authors propose SAFM (Sparse Adapter Fusion Method), which operates on every transformer layer of a decoder-only language model. In the first phase, called the decision stage, a warmup step is followed by a layer-by-layer architecture search. At each layer, the search chooses between inserting no adapter, reusing an earlier one, or creating a new one, and it favours the two cheaper options. In the second phase, called the tuning stage, the selected architecture is trained with pseudo-replay of earlier tasks. A layer-wise cosine-similarity loss also keeps the adapters in neighbouring layers distinct. Across several task orders, including E2ENLG and WikiSQL, SAFM matches or exceeds state-of-the-art baselines while using less than 60% of their parameters. This makes continual learning cheaper and more scalable.

Key points

  • Candidate adapters: At each layer, SAFM considers k+2 candidates: an empty adapter, the k distinct adapters already learned at that layer, and a newly initialized adapter. The layer's output is a weighted combination of these candidates, and the candidate with the highest weight is kept.
  • Backbone details: The main backbone is GPT-2, with 12 layers and a hidden size of 768. The authors also scale up to GPT-2-medium, GPT-2-large and Llama3-8B.
  • GPT-2 (124M) results, Order 1, after task 8: SAFM scores 56.55 with 6.7M learnable parameters, while ACM scores 56.11 with 14.3M.
  • Llama3-8B results, Order 1, after task 8: SAFM scores 42.78 with 1031.1M learnable parameters, while ACM scores 20.95 with 5279.3M. Both methods perform worse on Llama3-8B than on GPT-2, which the authors attribute to limited training data.
  • Further analysis: Leaving one adapter layer out of the architecture search improves results, and the gain is larger for higher layers.
  • Case study: On E2ENLG and WikiSQL examples, ACM drops input fields or produces wrong SQL, while SAFM's outputs match the reference.
  • Limitation: SAFM handles new information at the task level but not at the sample level, so it may perform worse on outlier samples.
  • Code: Available at github.com/OzymandiasChen/SAFM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Shenzhen Technology UniversityShenzhen · China3 works · 59 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Min Zeng, Xi Chen, Haiqin Yang and Yike Guo. 2026. Sparse Adapter Fusion for Continual Learning in NLP. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.37

@inproceedings{zeng2026sparse,
  title = {{Sparse Adapter Fusion for Continual Learning in NLP}},
  author = {Min Zeng, Xi Chen, Haiqin Yang and Yike Guo},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.37},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.37}
}

References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154. find
  2. Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6523–… find
  3. Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. arXiv preprint arXiv:1909.05358. the Americas Chapter of the Asso…
  4. Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1. find
  5. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arxiv. arXiv preprint arXiv:2203.15556.
  6. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML, pages 2790–2799. find
  7. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  8. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… find
  9. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In NIPS, volume 30. find
  10. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  11. Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021. Continual learning in task-oriented dialogue systems. In EMNLP. find
  12. Jorge A. Mendez and Eric Eaton. 2023. How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition. find
  13. Trans. Mach. Learn. Res., 2023. find
  14. Rich Caruana. 1997. Multitask learning. Machine learning, 28:41–75. find
  15. Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. 2020. Continual learning for natural language generation in task-oriented dialog systems. In EMNLP, pages 3461–3474. find
  16. Xi Chen and Min Zeng. 2025. Prototype conditioned generative replay for continual learning in nlp. In Proceedings of the 2025 Conference of the Nations of Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706…
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  19. (Volume 1: Long Papers), pages 663–677, Mexico City, Mexico. Association for Computational Linguistics. find
  20. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
  21. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. arXiv preprint arXiv:1905.08743.
  22. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence, 05, pages 8689–8696.
  23. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010. find
  24. Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & compress: A scalable framework for continual learning. In ICML, pages 4528–4537. find
  25. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. find
  26. Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329. find
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  28. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024a. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pattern Anal. Mach. Intell., 46(8):5362–5383. find
  29. Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10658–10671, Singapore. Association for Co… find
  30. Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. 2024b. InsCL: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computat… find
  31. Min Zeng, Haiqin Yang, Xi Chen, and Yike Guo. 2025. Task-wrapped continual learning in task-oriented dialogue systems. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3173–3183. find
  32. Min Zeng, Haiqin Yang, Wei Xue, Qifeng Liu, and Yike Guo. 2024. Dirichlet continual learning: Tackling catastrophic forgetting in nlp. In The 40th Conference on Uncertainty in Artificial Intelligence. find
  33. Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In ICML, pages 3987–3995. find
  34. Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2022. Continual sequence generation with adaptive compositional modules. In ACL. find
  35. Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024. SAPT: A shared attention framework for parameter-efficient continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Comput… find
  36. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
  37. Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking. In ACL, pages 1124–1137. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-sparse-adapter-fusion-continual-learning/