📄 OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

Paper · 2024 · Fuzhao Xue, Zian Zheng, Yao Fu et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations

Cite

Cite as Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You. 2024. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models. arXiv:2402.01739.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.01739 · arXiv 2402.01739

@misc{xue2024openmoe,
  title = {{OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models}},
  author = {Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You},
  year = {2024},
  doi = {10.48550/arxiv.2402.01739},
  eprint = {2402.01739},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01739}
}

References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Anil et al., “Palm 2 technical report,” arXiv preprint arXiv:2305.10403, 2023. Anonymous, “(inthe)wildchat: 570k chatGPT interaction logs in the wild,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=Bl8u7ZRlbM. M. A… ↗ our card
2. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y. Kan, Eds., Vancouver,… our card
3. J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1, 2019, p. 2. our card
4. D. Kocetkov et al., “The stack: 3 tb of permissively licensed source code,” Preprint, 2022. find
5. A. Komatsuzaki et al., “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=T5nUQDrM4u. ↗
6. D. Lepikhin et al., “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. find
7. M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in International Conference on Machine Learning, PMLR, 2021, pp. 6265–6274. find
8. J. Li, Z. Zhang, and H. Zhao, “Self-prompting large language models for open-domain qa,” arXiv preprint arXiv:2212.08635, 2022. — [arXiv:2212.08635](https://arxiv.org/abs/2212.08635) find
9. R. Li et al., “Starcoder: May the source be with you!” arXiv preprint arXiv:2305.06161, 2023. find
10. Y. Liu et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. find
11. Y. Lou, F. Xue, Z. Zheng, and Y. You, “Cross-token modeling with conditional computation,” arXiv preprint arXiv:2109.02008, 2021. — [arXiv:2109.02008](https://arxiv.org/abs/2109.02008) find
12. B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, “Multimodal contrastive learning with limoe: The language-image mixture of experts,” Advances in Neural Information Processing Systems, vol. 35, pp. 9564–9576, 2022. find
13. E. Nijkamp et al., “Xgen-7b technical report,” arXiv preprint arXiv:2309.03450, 2023. — [arXiv:2309.03450](https://arxiv.org/abs/2309.03450) find
14. J. Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby, “From sparse to soft mixtures of experts,” arXiv preprint arXiv:2308.00951, 2023. — [arXiv:2308.00951](https://arxiv.org/abs/2308.00951) find
15. J. W. Rae et al., “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446, 2021. find
16. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html. ↗ our card
17. C. Riquelme et al., “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583–8595, 2021. find
18. S. Roller, S. Sukhbaatar, J. Weston, et al., “Hash layers for large sparse models,” Advances in Neural Information Processing Systems, vol. 34, pp. 17 555–17 566, 2021. find
19. B. Roziere et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023. our card
20. N. Shazeer, “Glu variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020. find
21. N. Shazeer et al., “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. find
22. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. find
23. L. Soldaini et al., “Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research,” arXiv preprint, 2023. find
24. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing, vol. 568, p. 127 063, 2024. our card
25. Y. Tay et al., “Ul2: Unifying language learning paradigms,” in The Eleventh International Conference on Learning Representations, 2022. our card
26. Y. Tay et al., “Unifying language learning paradigms,” arXiv preprint arXiv:2205.05131, 2022. our card
27. L.-M. Team, Llama-moe: Building mixture-of-experts from llama with continual pre-training, Dec. 2023. [Online]. Available: https://github.com/pjlab-sys4nlp/llama-moe. ↗ our card
28. H. Touvron et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. our card
29. B. Wang and A. Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, https://github.com/kingoflolz/mesh-transformer-jax, May 2021. ↗
30. G. Wenzek et al., “CCNet: Extracting high quality monolingual datasets from web crawl data,” English, in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari et al., Eds., Marseille, France: European Language Resources Association, May 2020, pp. 4003–4012, ISBN : 979… our card
31. Y. Xu et al., “Gspmd: General and scalable parallelization for ml computation graphs,” arXiv preprint arXiv:2105.04663, 2021. — [arXiv:2105.04663](https://arxiv.org/abs/2105.04663) find
32. F. Xue, X. He, X. Ren, Y. Lou, and Y. You, “One student knows all experts know: From sparse to dense,” arXiv preprint arXiv:2201.10890, 2022. — [arXiv:2201.10890](https://arxiv.org/abs/2201.10890) find
33. F. Xue, Z. Shi, F. Wei, Y. Lou, Y. Liu, and Y. You, “Go wider instead of deeper,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8779–8787. find
34. P. Yu et al., “Efficient language modeling with sparse all-mlp,” arXiv preprint arXiv:2203.06850, 2022. — [arXiv:2203.06850](https://arxiv.org/abs/2203.06850) find
35. P. Zhang, G. Zeng, T. Wang, and W. Lu, Tinyllama: An open-source small language model, 2024. arXiv: 2401.02385 [cs.CL]. our card
36. Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, “Deep long-tailed learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. find
37. L. Zheng et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” arXiv preprint arXiv:2306.05685, 2023. find
38. Y. Zhou et al., “Brainformers: Trading simplicity for efficiency,” in International Conference on Machine Learning, PMLR, 2023, pp. 42 531–42 542. find
39. Y. Zhou et al., “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems, vol. 35, pp. 7103–7114, 2022. find
40. B. Zoph et al., “St-moe: Designing stable and transferable sparse expert models,” URL https://arxiv.org/abs/2202.08906, 2022. ↗

Updated 24 October 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/