Elamur Research⌕
☰
home / papers / 2026 / Mixture of Heterogeneous Grouped Experts for Language Modeling

Mixture of Heterogeneous Grouped Experts for Language Modeling

Paper · 2026 · Zhicheng Ma, Xiang Liu, Zhaoxiang Liu et al.

Summary

Standard Mixture-of-Experts (MoE) language models use experts of one fixed size, so every token receives the same compute regardless of how hard it is to predict. Earlier heterogeneous designs (MoDSE, HMoE) vary expert sizes, but they either route tokens poorly or overload some GPUs. This industry-track paper from China Unicom proposes MoHGE, which arranges experts into groups. Experts within a group share a size, and sizes differ between groups.

A two-level router first scores groups against learned centroids and keeps the top ones. It then chooses experts globally among those groups and renormalizes their weights. Three additions handle efficiency and balance:

  • a Group-Wise Auxiliary Loss mildly penalizes routing to large groups, so easy tokens lean toward small experts;
  • an All-size Group-decoupling Allocation places members of every group on all GPUs;
  • an Intra-Group Experts Auxiliary Loss evens out usage inside each group.

MoHGE matches standard MoE quality with about 20% fewer total parameters while keeping GPU load even. This makes heterogeneous experts practical for deployments where cost matters.

Key points

  • MoHGE uses about 20% fewer total parameters than standard MoE baselines while scoring comparably or slightly better. It also beats the reproduced heterogeneous models on most benchmarks.
  • In a 14B-scale test, the i-th expert of each of 8 size groups was placed on the i-th of 8 GPUs. Per-GPU routing shares stayed near 12.5% (range 11.9–13.0%), with standard deviations of roughly 0.002–0.003 per group.
  • Token difficulty was measured two ways: frequency rank (Top 1K, 1K–5K, 5K–10K, Beyond 10K) and perplexity bins (≤5, ≤10, >10). The smallest group (Group 1) received 16.3% of Top-1K tokens but only 11.3% of rare tokens. The largest group (Group 8) showed the reverse pattern, 9.7% versus 13.3%.
  • At 3B scale, MoHGE-3B was compared with MoDSE-3B and HMoE-3B. On GSM8K it scored 4.02 versus 3.71 and 3.94, and on LAMBADA 62.37 versus 61.98 and 62.25. Margins are small, and HMoE slightly leads on SIQA (35.62 vs 35.57).
  • Only HMoE showed unbalanced GPU utilization, which the authors see as a barrier to scaling it up.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢China United Network Communications GroupBeijing · China2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi and Shiguo Lian. 2026. Mixture of Heterogeneous Grouped Experts for Language Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.20

@inproceedings{ma2026mixture,
  title = {{Mixture of Heterogeneous Grouped Experts for Language Modeling}},
  author = {Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi and Shiguo Lian},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.20},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.20}
}

References

25 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  4. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
  7. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  8. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  9. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, and 1 others. 2024a. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434. find
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024b. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  12. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. find
  13. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2024. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. ↗
  16. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume… find
  17. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031.
  18. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
  19. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. find
  20. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. find
  21. Manxi Sun, Wei Liu, Jian Luan, Pengzhi Gao, and Bin Wang. 2024. Mixture of diverse size experts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1608–1621, Miami, Florida, US. Association for Computational Linguistics. find
  22. Neil C Thompson, Kristjan Greenewald, Keeheon Lee, Gabriel F Manso, and 1 others. 2020. The computational limits of deep learning. arXiv preprint arXiv:2007.05558, 10. — [arXiv:2007.05558](https://arxiv.org/abs/2007.05558) find
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  24. An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, JN Han, Zhanhui Kang, Di Wang, and 1 others. 2024. Hmoe: Heterogeneous mixture of experts for language modeling. arXiv preprint arXiv:2408.10681.
  25. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, and 1 others. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-mixture-heterogeneous-grouped-experts-language/