Elamur Research⌕
☰
home / papers / 2025 / Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs

Paper · 2025 · Ling Team; Binwei Zeng; Chao Huang; Chao Zhang; Changxin Tian; Cong Chen; Dingnan Jin; Feng Yu; Feng Zhu; Feng Yuan; Fakang Wang; Gangshan Wang; Guangyao Zhai; Haitao Zhang; Huizhong Li; Jun Zhou; Jia Liu; Junpeng Fang; Junjie Ou; Jun Hu; Ji Luo; Ji Zhang; Jian Liu; Jian Sha; Jianxue Qian; Jiewei Wu; Junping Zhao; Jianguo Li; Jubao Feng; Jingchao Di; Junming Xu; Jinghua Yao; Kuan Xu; Kewei Du; Longfei Li; Lei Liang; Lu Yu; Li Tang; Lin Ju; Peng Xu; Qing Cui; Song Liu; Shicheng Li; Shun Song; Song Yan; Tengwei Cai; Tianyi Chen; Ting Guo; Ting Huang; Tao Feng; Tao Wu; Wei Wu; Xiaolu Zhang; Xueming Yang; Xin Zhao; Xiaobo Hu; Xin Lin; Yao Zhao; Yilong Wang; Yongzhen Guo; Yuanyuan Wang; Yue Yang; Yang Cao; Yuhao Fu; Yi Xiong; Yanzhe Li; Zhe Li; Zhiqiang Zhang; Ziqi Liu; Zhaoxin Huan; Zujie Wen; Zhenhang Sun; Zhuoxuan Du; and Zhengyu He

Summary

This technical report from Ant Group's Ling Team asks whether frontier-scale Mixture-of-Experts models can be trained without scarce top-tier accelerators such as H100/H800. The team built two open MoE LLMs: Ling-Lite (16.8B total, 2.75B active parameters) and Ling-Plus (290B total, 28.8B active). Ling-Plus was pre-trained on about 9 trillion tokens spread across five heterogeneous, lower-spec hardware configurations.

Their engineering stack has four parts:

  • Architecture and training: scaling-law-guided architecture choice, a unified framework (DLRover) and asynchronous training (EDiT).
  • Infrastructure: storage and I/O optimizations, plus multi-level anomaly detection with automatic checkpoint recovery.
  • Evaluation: a cross-cluster evaluation pipeline.
  • Data: tool-use training data synthesized from knowledge graphs.

The resulting models perform roughly on par with similar-size dense and MoE systems, such as Qwen2.5-72B-Instruct and DeepSeek-V2.5-1210-Chat, and are especially strong at tool use. Pre-training costs about 20% less than on premium hardware. The work matters because it shows that resource-constrained labs can train large models on cheaper, mixed hardware. Its main caveat is that operators and frameworks must be carefully aligned across platforms.

Key points

  • Training cost: processing 1T tokens costs roughly 6.35M RMB on the high-end device D versus about 5.08M RMB on lower-spec hardware, a saving of nearly 20%.
  • Tooling gains:
    • XPUTimer cuts profiling memory by about 90%.
    • EDiT shortens training time by up to 66.1%.
    • Storage/training co-design halves I/O overhead in MoE runs.
  • Evaluation consistency: the Flood-based evaluation system keeps cross-cluster results within about 0.5% average deviation.
  • Cross-platform alignment: stable training required router z-loss and related fixes. Moving between platforms also required aligning both the forward and backward passes, covering matmul, attention, MLP, router gradients and tensor parallelism.
  • Availability: the models are released on Hugging Face under inclusionAI, and a coder model is planned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksGPU & parallel computing · 19 works

Cite

Cite as Ling Team; Binwei Zeng; Chao Huang; Chao Zhang; Changxin Tian; Cong Chen; Dingnan Jin; Feng Yu; Feng Zhu; Feng Yuan; Fakang Wang; Gangshan Wang; Guangyao Zhai; Haitao Zhang; Huizhong Li; Jun Zhou; Jia Liu; Junpeng Fang; Junjie Ou; Jun Hu; Ji Luo; Ji Zhang; Jian Liu; Jian Sha; Jianxue Qian; Jiewei Wu; Junping Zhao; Jianguo Li; Jubao Feng; Jingchao Di; Junming Xu; Jinghua Yao; Kuan Xu; Kewei Du; Longfei Li; Lei Liang; Lu Yu; Li Tang; Lin Ju; Peng Xu; Qing Cui; Song Liu; Shicheng Li; Shun Song; Song Yan; Tengwei Cai; Tianyi Chen; Ting Guo; Ting Huang; Tao Feng; Tao Wu; Wei Wu; Xiaolu Zhang; Xueming Yang; Xin Zhao; Xiaobo Hu; Xin Lin; Yao Zhao; Yilong Wang; Yongzhen Guo; Yuanyuan Wang; Yue Yang; Yang Cao; Yuhao Fu; Yi Xiong; Yanzhe Li; Zhe Li; Zhiqiang Zhang; Ziqi Liu; Zhaoxin Huan; Zujie Wen; Zhenhang Sun; Zhuoxuan Du; and Zhengyu He. 2025. Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs. arXiv:2503.05139.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.05139 · arXiv 2503.05139

@misc{du2025every,
  title = {{Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs}},
  author = {Ling Team; Binwei Zeng; Chao Huang; Chao Zhang; Changxin Tian; Cong Chen; Dingnan Jin; Feng Yu; Feng Zhu; Feng Yuan; Fakang Wang; Gangshan Wang; Guangyao Zhai; Haitao Zhang; Huizhong Li; Jun Zhou; Jia Liu; Junpeng Fang; Junjie Ou; Jun Hu; Ji Luo; Ji Zhang; Jian Liu; Jian Sha; Jianxue Qian; Jiewei Wu; Junping Zhao; Jianguo Li; Jubao Feng; Jingchao Di; Junming Xu; Jinghua Yao; Kuan Xu; Kewei Du; Longfei Li; Lei Liang; Lu Yu; Li Tang; Lin Ju; Peng Xu; Qing Cui; Song Liu; Shicheng Li; Shun Song; Song Yan; Tengwei Cai; Tianyi Chen; Ting Guo; Ting Huang; Tao Feng; Tao Wu; Wei Wu; Xiaolu Zhang; Xueming Yang; Xin Zhao; Xiaobo Hu; Xin Lin; Yao Zhao; Yilong Wang; Yongzhen Guo; Yuanyuan Wang; Yue Yang; Yang Cao; Yuhao Fu; Yi Xiong; Yanzhe Li; Zhe Li; Zhiqiang Zhang; Ziqi Liu; Zhaoxin Huan; Zujie Wen; Zhenhang Sun; Zhuoxuan Du; and Zhengyu He},
  year = {2025},
  doi = {10.48550/arxiv.2503.05139},
  eprint = {2503.05139},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.05139}
}

References

86 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ryoko AI. Sharegpt dataset, 2023. URL https://huggingface.co/datasets/RyokoAI/ShareGPT52K. ↗
  2. Alipay. Alipay: https://www.alipay.com/, 2025. URL https://www.alipay.com/. ↗
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  5. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. URL https://arxiv.org/abs/1607.04606. ↗
  7. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022. find
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  9. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint arXiv:2312.14033, 2023. — [arXiv:2312.14033](https://arxiv.org/abs/2312.14033) find
  10. Jialiang Cheng, Ning Gao, Yun Yue, Zhiling Ye, Jiadi Jiang, and Jian Sha. EDiT: A local-SGD-based efficient distributed training method for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xtlMtbVfWu. ↗
  11. Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pages 4057–4086. PMLR, 2022. find
  12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  13. Claude. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ModelCardClaude3.pdf. ↗
  14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  15. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024.
  16. Weihao Cui, Ji Zhang, Han Zhao, Chao Liu, Wenhao Zhang, Jian Sha, Quan Chen, Bingsheng He, and Minyi Guo. Xputimer: Anomaly diagnostics for divergent llm training in gpu clusters of thousand-plus scale. arXiv preprint arXiv:2502.05413, 2025. find
  17. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. find
  18. DeepSeek. Deepseek opensourceweek, 2025. URL https://github.com/deepseek-ai/open-infra-index/tree/main/202502OpenSourceWeek. ↗
  19. DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism, 2024a. URL https://arxiv.org/abs/2401.02954. ↗
  20. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024b. URL https://arxiv.org/abs/2405.04434. ↗
  21. DeepSeek-AI. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437. ↗
  22. DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie,… — DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
  23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805. ↗
  24. DLRover. Dlrover: An automatic distributed deep learning system, 2023. URL https://github.com/intelligent-machine-learning/dlrover. ↗
  25. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  26. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. find
  27. FlagOpen. Flagscale: a large model toolkit based on open-sourced projects, 2025. URL https://github.com/FlagOpen/FlagScale. ↗
  28. Flood. Flood: A toolkit for llm painless inference acceleration, 2025. URL https://github.com/alipay/PainlessInferenceAcceleration. ↗
  29. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. find
  30. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127, 2024. find
  31. Gemini. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530. ↗
  32. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024.
  33. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140, 2024.
  34. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  35. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  36. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
  37. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  38. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991–63010, 2023.
  39. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  40. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  41. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  42. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180. ↗
  43. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017.
  44. LangChain. Langchain react: https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/, 2025. URL https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/. ↗
  45. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. find
  46. Chenliang Li, Hehong Chen, Ming Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu, Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, Hongzhu Shi, Ji Zhang, Fei Huang, and Jingren Zhou. Modelscope-agent: Building your customizable agent system with open-source large language models, 2023a. URL https://ar… find
  47. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023b.
  48. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024.
  49. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
  50. Hongzhi Luan, Changxin Tian, Zhaoxin Huan, Xiaolu Zhang, Kunlong Chen, Zhiqiang Zhang, and Jun Zhou. Toward stable and consistent evaluation results: A new methodology for base model evaluation, 2025. URL https://arxiv.org/abs/2503.00812. — [arXiv:2503.00812](https://arxiv.org/abs/2503.00812) ↗
  51. MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024, February 2024., 2024. URL URLhttps://maa.org/math-competitions/american-invitational-mathematics-examination-aime. find
  52. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–15018, 20…
  53. MiniMax. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https://arxiv.org/abs/2501.08313. ↗
  54. MTEB. Mteb leaderboard, 2024. URL https://huggingface.co/spaces/mteb/leaderboard. ↗
  55. OpenAI. Gpt-4 technical report, 2024a. URL https://arxiv.org/abs/2303.08774. ↗
  56. OpenAI. Introducing simpleqa, 2024b. URL URLhttps://openai.com/index/introducing-simpleqa/. find
  57. OpenAI. Openai function calling url: https://platform.openai.com/docs/guides/function-calling, 2025. URL https://platform.openai.com/docs/guides/function-calling. ↗
  58. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
  59. Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In NeurIPS, 2024.
  60. PurpleLlama. Llama-guard3 url: https://github.com/meta-llama/purplellama/blob/main/llama-guard3, 2024. URL https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3. ↗
  61. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
  62. Qwen. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. ↗
  63. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
  64. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning, pages 18332–18346.…
  65. RapidAPI. Rapidapi: https://rapidapi.com/, 2025. URL https://rapidapi.com/. ↗
  66. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  67. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023.
  68. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  69. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. ↗
  70. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053. ↗
  71. Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Chengming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, Connor Holmes, Martin Cai, Adam Ghanem, Zhongzhu Zhou, Yuxiong He, Pete Luferenko, Divya Kumar, Jonathan Weyn, Ruixiong Zhang, Sylwester Klocek, Vol… — [arXiv:2310.04610](https://arxiv.org/abs/2310.04610) find
  72. Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. Nexusraven: a commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. find
  73. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  74. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023.
  75. Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. Code less, align more: Efficient llm fine-tuning for code generation with data pruning. CoRR, 2024. find
  76. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datase…
  77. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120, 2023. find
  78. Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705, 2023.
  79. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024.
  80. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard, 2024. URL https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html. ↗
  81. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023.
  82. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… find
  83. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  84. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023.
  85. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
  86. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.08906. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ling-team-2025-every-flop-counts-scaling/