大模型时代的混合专家系统优化综述
Summary
Mixture of experts (MoE) has become a leading architecture for large models. Because only a few experts are activated per input, parameter counts can grow while compute rises only sublinearly. At scale, however, MoE systems face four bottlenecks:
- heavy memory demand;
- costly All-to-All exchanges between devices;
- wasted computation from zero-padding and uneven expert load;
- difficulty picking a parallel strategy that suits constantly shifting workloads.
This Chinese-language survey, by authors at IEIT SYSTEMS/Inspur, collects recent work on these system-side problems. It first explains core MoE ideas: Top-k gating, expert capacity, expert parallelism, and how MoE differs from classic ensemble learning. It then traces model development from the sparsely gated LSTM-based MoE through GShard, Switch Transformers, Mixtral-8x7B and DeepSeekMoE. Next, it sorts optimization techniques into four groups: memory, communication, computation and parallel scaling. For each, it compares the main ideas, suitable scenarios and remaining weaknesses, and it ends with future research directions. No new dataset or model is introduced. Its value is that, by the authors' account, it is the first review focused on MoE systems engineering rather than algorithms.
Key points
- Taxonomy of techniques:
- Memory: offloading to CPU memory/SSD; parameter compression.
- Communication: hierarchical communication; redundancy elimination; topology-aware routing.
- Computation: dynamic capacity; expert/device load balancing; statistics- and ML-based expert activation prediction; static vs. adaptive pipelining; kernel optimization.
- Parallel scaling: static vs. adaptive parallelism.
- Memory gap: top-end GPU memory stayed at 80 GB during 2021–2023. Switch Transformers already had 1.6 trillion parameters in 2021.
- GLaM: 1.2T parameters, about 7× GPT-3, yet needs roughly a third of its compute and outperforms GPT-3 on 29 NLP benchmarks.
- GShard: introduced expert capacity and expert parallelism, and built a 600B-parameter MoE Transformer for translation.
- Source: Journal of Computer Research and Development 62(5):1164–1189, 2025; DOI 10.7544/issn1000-1239.202440016.
Cite
@article{2025,
title = {{大模型时代的混合专家系统优化综述}},
author = {史宏志 and 赵健 and 赵雅倩 and 李茹杨 and 魏辉 and 胡克坤 and 温东超 and 金良},
journal = {计算机研究与发展 (Journal of Computer Research and Development)},
year = {2025},
doi = {10.7544/issn1000-1239.202440016},
url = {https://doi.org/10.7544/issn1000-1239.202440016}
}References
- experts[J]. arXiv preprint, arXiv: 2407.06204, 2024 — [arXiv:2407.06204](https://arxiv.org/abs/2407.06204) find
- ［36］ Lepikhin D, Lee H, Xu Yuandong, et al. GShard: Scaling giant models with conditional computation and automatic sharding [J]. arXiv preprint, arXiv: 2006.16668, 2020 find
- ［37］ Zhang Zhengyan, Lin Yankai, Liu Zhiyuan, et al. MoEfication: Transformer feed-forward layers are mixtures of experts[J]. arXiv preprint, arXiv: 2110.01786, 2021 find
- ［38］ Zuo Simiao, Zhang Qingru, Liang Chen, et al. MoEBERT: From BERT to mixture-of-experts via importance-guided adaptation[J]. arXiv preprint, arXiv: 2204.07675, 2022 find
- ［39］ Zhu Tong, Qu Xiaoye, Dong Daize, et al. LLaMA-MoE: Building mixture-of-experts from LLaMA with continual pre-training[J]. arXiv preprint, arXiv: 2406.16554, 2024 find
- ［40］ Dai Damai, Deng Chenqi, Zhao Chenggang, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models[J]. arXiv preprint, arXiv: 2401.06066, 2024 find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2025-system-optimization-mixture/