📄 HMoE: Heterogeneous Mixture of Experts for Language Modeling

Summary

Standard Mixture of Experts (MoE) language models use experts of identical size, even though some tokens are harder to process than others. The authors argue this uniformity limits how much experts specialize and wastes parameters. They propose HMoE, in which the FFN experts in a layer have different hidden dimensions and therefore different capacities. They also tested a naive version with no auxiliary objective, and it did not beat homogeneous MoE: routing favored the large experts and left the small ones mostly unused. To fix this, they add the P-Penalty Loss, a training term that pushes the router toward smaller experts. They evaluate HMoE with both Top-K and Top-P routing and compare several schemes for choosing expert sizes. In the reported pre-training experiments, HMoE reaches lower loss than conventional MoE while activating fewer parameters, and it scores better on downstream benchmarks. The work matters because it shows expert size is a useful design dimension for sparse LLMs, giving better accuracy at lower compute.

Key points

• Experts follow the LLaMa-style SiLU-gated FFN. Heterogeneity comes only from varying each expert's hidden size, because mixing different expert architectures made training unstable.
• The P-Penalty Loss promotes activation of small experts and outperforms the standard load balancing loss for HMoE. Without guidance, HMoE (e.g., relative sizes {9, 11, …, 23} for experts a–h) underperforms homogeneous MoE.
• Among expert-size distributions, a hybrid mix of similar and varied sizes worked best, ahead of arithmetic. Geometric was worst because its smallest experts were too weak and seldom chosen.
• Wasserstein-distance heatmaps show more differentiated experts in HMoE than in homogeneous MoE. In HMoE, experts of similar size form clusters, while homogeneous experts split into roughly two groups.
• Rare or ambiguous tokens are routed more often to larger experts. Code is available at github.com/AnWang-AI/HMoE (EMNLP 2025, Tencent Hunyuan et al.).