🔗 Who cites «Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints»

86 papers of our library cite it (of the 149,439 whose bibliographies we read) · 19 with a page here, the most cited first · ← back to «Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints»

1. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsConference on Empirical Methods in Natural Language Processing (EMNLP)Joshua Ainslie, James Lee-Thorp, Michiel de Jong, et al.2023 · 1,026 citations
2. Crosslingual Generalization through Multitask FinetuningarXiv:2211.01786Niklas Muennighoff, Thomas Wang, Lintang Sutawika, et al.2022 · 703 citations
3. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training StrategiesarXiv:2404.06395Shengding Hu, Yuge Tu, Xu Han, et al.2024 · 269 citations
4. A Survey on Efficient Inference for Large Language ModelsarXiv:2404.14294Zixuan Zhou, Xuefei Ning, Ke Hong, et al.2024 · 113 citations
5. Arcee's MergeKit: A Toolkit for Merging Large Language ModelsarXiv:2403.13257Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, et al.2024 · 105 citations
6. Retrieval Head Mechanistically Explains Long-Context FactualityarXiv:2404.15574Wenhao Wu, Yizhong Wang, Guangxuan Xiao, et al.2024 · 94 citations
7. Accelerating Distributed MoE Training and Inference with LinaarXiv:2210.17223Jiamin Li, Yimin Jiang, Yibo Zhu, et al.2022 · 92 citations
8. LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-TrainingConference on Empirical Methods in Natural Language Processing (EMNLP)Tong Zhu, Xiaoye Qu, Daize Dong, et al.2024 · 85 citations
9. SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-ScalingarXiv:2312.15166Dahyun Kim, Chanjun Park, Sanghoon Kim, et al.2023 · 77 citations
10. Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language ModelsarXiv:2305.14705Sheng Shen, Le Hou, Yanqi Zhou, et al.2023 · 70 citations
11. Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction TuningarXiv:2309.05444Ted Zadouri, Ahmet Ustun, Arash Ahmadian, et al.2023 · 69 citations
12. OpenMoE: An Early Effort on Open Mixture-of-Experts Language ModelsarXiv:2402.01739Fuzhao Xue, Zian Zheng, Yao Fu, et al.2024 · 64 citations
13. Stable LM 2 1.6B Technical ReportarXiv:2402.17834Marco Bellagente, Jonathan Tow, Dakota Mahan, et al.2024 · 46 citations
14. Towards the Law of Capacity Gap in Distilling Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Chen Zhang, Qiuchi Li, Dawei Song, et al.2025 · 27 citations
15. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective DecodingarXiv:2507.19427StepFun, Bin Wang, Bojun Wang, et al.2025 · 26 citations
16. FuseChat: Knowledge Fusion of Chat ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Fanqi Wan, Longguang Zhong, Ziyi Yang, et al.2025 · 25 citations
17. Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language ModelsarXiv:2406.06563Tianwen Wei, Bo Zhu, Liang Zhao, et al.2024 · 21 citations
18. Analytical FFN-to-MoE Restructuring via Activation Pattern AnalysisAnnual Meeting of the Association for Computational Linguistics (ACL)Zehua Pei, Hui-Ling Zhen, Lancheng Zou, et al.2026 · 1 citation
19. A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔ Integration into Upcycled MoEAnnual Meeting of the Association for Computational Linguistics (ACL)Hao Zhou, Tianhao Li, Zhijun Wang, et al.2026

67 more papers of our library cite it; they have no page here yet.

This page in Markdown → https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/cited-by/