Elamur Research⌕
☰
home / papers / 2026 / Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models

Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models

Paper · 2026 · Yuanbo Tang, Naifan Zhang, Yan Tang et al.

Summary

Mixture-of-Experts LLMs send each token to a few expert subnetworks, but it remains unclear whether individual experts really specialize. Drawing on the neuroscience idea that functions are carried by neural populations, the authors study groups of experts that activate together across layers. They aggregate token routing weights into sentence-level activation matrices for MMLU-pro inputs. These matrices are factorized with Hierarchical Sparse Dictionary Learning (HSDL), a multi-level extension of an existing Bayesian dictionary learning method, which yields collaboration patterns from coarse to fine. GPT-4 labels the patterns, and they correspond to capabilities like solving math problems or handling sentence structure. Groups are more domain-focused than single experts. The authors then use these groups for expert pruning (CAEP), keeping functionally important combinations. On DeepSeek-MoE and Phi-MoE, CAEP outperforms SEER-MoE and GEM by about 2.5% on average and still leads when half of the experts are removed. The work gives an interpretability lens on MoE models, a cheap compression recipe, and an interactive visualization platform.

Key points

  • Analysis data: 2,812 MMLU-pro samples covering mathematics, computer science, physics, law and psychology; the models studied are DeepSeek-MoE (16B) and Phi-MoE.
  • With 25% of experts pruned, CAEP achieves the following average scores (OBQA, ARC-C, HellaSwag, WinoGrande, RTE):
    • DeepSeek: 0.650, versus 0.628 for GEM, 0.626 for SEER-MoE and 0.692 for the original model.
    • Phi-MoE: 0.652, versus 0.636 for GEM.
  • With 50% of experts pruned on DeepSeek-MoE, CAEP scores 0.552, compared with 0.529 for GEM and 0.512 for SEER-MoE.
  • Domain ablations confirm the groups matter:
    • Removing the math-specific expert group lowers math accuracy by 21.7%.
    • Keeping only domain-relevant experts costs just 5.7%.
    • Related domains (Math/Physics/CS) affect each other more than distant ones (Law/Physics).
  • HSDL adds roughly 1,230 TFLOPs, about 2 minutes on one GTX 1080 Ti. Pruning 25% of experts saves 3.7B parameters (14.7GB). The semantic labels come from GPT-4 and have not been checked by humans.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao and Yang Li. 2026. Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.104

@inproceedings{tang2026specialization,
  title = {{Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models}},
  author = {Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao and Yang Li},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.104},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.104}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A Survey on Mixture of Experts. arXiv preprint. find
  2. Chen Chen, Hao Su, Qixing Huang, Lin Zhang, and Leonidas Guibas. 2013. Pathlet learning for compressing and planning trajectories. In Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 392–395, Orlando Florida. ACM. find
  3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  4. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Mo… find
  5. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  6. Leo Gao, Gabriel Goh, and Ilya Sutskever. Scaling and evaluating sparse autoencoders. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Pha… find
  7. Shwai He, Daize Dong, Liang Ding, and Ang Li. 2024. Demystifying the Compression of Mixture-of-Experts Through a Unified Framework. arXiv preprint. find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  9. Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. 2020. Visual Compositional Learning for Human-Object Interaction Detection. In Computer Vision – ECCV 2020, pages 584–600, Cham. Springer International Publishing. find
  10. Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. 2021. Detecting Human-Object Interaction via Fabricated Compositional Learning. pages 14646–14655. find
  11. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  12. Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2023. Merge, then compress: Demystify efficient smoe with hints from its routing policy. CoRR. find
  13. Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2024. A Closer Look into Mixture-of-Experts in Large Language Models. arXiv preprint. find
  14. Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. CoRR, abs/2402.14800.
  15. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. find
  16. Alexandre Muzio, Alex Sun, and Churan He. 2024. SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts. arXiv preprint. find
  17. Stefano Panzeri, Monica Moroni, Houman Safaai, and Christopher D. Harvey. 2022. The structures and functions of correlations in neural population codes. Nature Reviews Neuroscience, 23(9):551–567. find
  18. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024. Improving Dictionary Learning with Gated Sparse Autoencoders. arXiv preprint.
  19. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  20. Yuanbo Tang, Zhiyuan Peng, and Yang Li. 2023. Explainable Trajectory Representation through Dictionary Learning. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’23, pages 1–4, New York, NY, USA. Association for Computing Machinery. find
  21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.
  22. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Yi Ma. 2009. Robust Face Recognition via Sparse Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2(31):210–227. find
  23. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. In Forty-first International Conference on Machine Learning.
  24. Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. 2010. Image super-resolution via sparse representation. IEEE transactions on image processing: a publication of the IEEE Signal Processing Society, 19(11):2861–2873. find
  25. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  26. Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. 2024. Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts. arXiv preprint. find
  27. Zihui Zhao, Yuanbo Tang, Jieyu Ren, Xiaoping Zhang, and Yang Li. 2025. A Unified Probabilistic Framework for Dictionary Learning with Parsimonious Activation. arXiv preprint. ArXiv:2509.25690 [cs]. — [arXiv:2509.25690](https://arxiv.org/abs/2509.25690) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2026-specialization-through-collaboration-understanding-expert/