Elamur Research⌕
☰
home / papers / 2026 / A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding

A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding

Paper · 2026 · Zhe Yang, Yi Huang, Yaqin Chen et al.

Summary

Labeled data for domain-specific natural language understanding is scarce. Multi-task learning can help, but current sparse designs have weaknesses. Skill-Net needs task-to-skill assignments chosen by hand. MMoE uses a fixed top-k routing that can overload some experts. The authors replace each feed-forward layer of a BERT-style encoder with a cluster of FFN "skill" modules. Tasks and skills each get their own learnable embeddings. A projected dot product followed by a sigmoid scores how relevant each skill is to each task. A smooth surrogate of a hard threshold turns these scores into near-binary on/off choices, so the whole model trains end to end. Each task can switch on a different number of skills. An orthogonality penalty keeps the skill embeddings distinct from one another. The method was tested on MacBERT with six Chinese NLU benchmarks and reached the best average F1. Its biggest gains were on token-level tasks. On three sentence-level datasets, however, single-task fine-tuning still scored higher. The main practical benefit is that task-skill mappings are learned rather than designed manually.

Key points

  • Setup: MacBERT backbone with 7 skill FFNs per layer; MMoE and Skill-Net baselines also use 7 routines.
  • Datasets: TNEWS, ChnSentiCorp, AFQMC, OCNLI, OntoNotesEE (NER) and CMRC2018 (span-extraction reading comprehension).
  • Average F1: 66.40, compared with Skill-Net 65.82, single-task fine-tuning 66.01, MT-DNN 63.89 and MMoE 63.34.
  • Largest gain: on OntoNotesEE the model scores 60.32 F1 against 44.92 for MMoE. It also leads on OCNLI and CMRC2018.
  • Ablation: setting the orthogonality weight β to 0 lowers the average to 65.51 (−0.89). The architecture, not the penalty, accounts for most of the improvement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Zhe Yang, Yi Huang, Yaqin Chen, Mengfei Guo, Xiaoting Wu and Junlan Feng. 2026. A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1886

@inproceedings{yang2026learnable,
  title = {{A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding}},
  author = {Zhe Yang, Yi Huang, Yaqin Chen, Mengfei Guo, Xiaoting Wu and Junlan Feng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1886},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1886}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations. find
  2. Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. 2024. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9). find
  3. Shijie Chen, Yu Zhang, and Qiang Yang. 2024. Multi-task learning in natural language processing: An overview. ACM Comput. Surv., 56(12). find
  4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 657–668, Online. Association for Computational Linguistics.
  5. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for Chinese machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf…
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. OCNLI: Original Chinese Natural Language Inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3512–3526, Online. Association for Computational Linguistics.
  8. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496, Florence, Italy. Association for Computational Linguistics…
  9. Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1930–1939, New York,… find
  10. Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, and Soroush Vosoughi. 2021. GradTS: A gradient-based automatic auxiliary task selection method based on transformer networks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5621–5632, Online and Punta Can… find
  11. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Ponti. 2023. Modular deep learning. Transactions on Machine Learning Research. Survey Certification. find
  12. Jonathan Pilault, Amine El hattami, and Christopher Pal. 2021. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. In International Conference on Learning Representations. find
  13. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. find
  14. Songbo Tan. 2020. Chnsenticorp. find
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur…
  16. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, and 1 others. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23(170):20. find
  17. Liang Xu and et al. 2020. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  18. Fan Zhang, Duyu Tang, Yong Dai, Cong Zhou, Shuangzhi Wu, and Shuming Shi. 2022. Skillnet-nlu: A sparsely activated model for general-purpose natural language understanding. find
  19. Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. 2023. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pag… find
  20. Hao Zhao, Jie Fu, and Zhaofeng He. 2023. Prototype-based HyperAdapter for sample-efficient multi-task tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4603–4615, Singapore. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-learnable-skill-combination-strategy-multi/