Elamur Research⌕
☰
home / papers / 2025 / A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

Paper · 2025 · Junyu Luo, Bohan Wu, Xiao Luo et al.

Summary

Post-training is the stage where LLMs acquire instruction following, alignment, reasoning and domain skills. It is constrained by three data problems: costly human annotation, shrinking gains from simply adding more examples, and static datasets that cannot track changing knowledge. This ACL 2025 survey by Luo et al. reviews ways to get more out of less data during LLM post-training. It approaches the question from the data side rather than through model architecture or compute. The authors structure prior work around a "data value flywheel" made up of five interacting stages: choosing which data to use, improving its quality, generating synthetic examples, distilling and compressing data, and building self-evolving data ecosystems. Each stage is divided into finer subcategories with representative methods. The survey compares these families on their reliance on source data, compute and model capacity, and closes with open problems. Its central claim is that efficiency gains come from extracting value throughout the data lifecycle, not from scaling volume. For teams with small annotation budgets, such as low-resource language projects, it provides a structured map of cost-saving techniques.

Key points

  • Taxonomy:
    • Selection: static filtering, dynamic selection, agent strategy, labeling efficiency.
    • Quality enhancement: semantic rewriting, toxicity control, distribution stabilization.
    • Synthesis: instruction-driven, knowledge-guided, adversarial.
    • Distillation: model, data, joint compression.
    • Self-evolution: self-iterative optimization, dynamic evaluation feedback, LLM-as-a-Judge.
  • Concrete example: Alpagasus matches full-data results using only about 17% of the original instruction data, selected by filtering. DeepSeek-R1 is cited as evidence that data-efficient post-training pays off.
  • Trade-offs (Table 1):
    • Selection is data-efficient but requires good raw data.
    • Synthetic generation and self-evolving methods need more compute and stronger models, but depend less on manual labels.
    • Distillation relies heavily on teacher capability.
  • Representative methods covered: LESS, Magpie, LLMLingua-2, Self-Rewarding Language Models, Self-Refine, JudgeLM, CalibraEval, TOXIGEN.
  • Future directions and resource: The authors highlight domain-specific data synthesis, scalable generation pipelines, and standardized metrics for synthetic data quality. A curated paper list is available at github.com/luo-junyu/Awesome-Data-Efficient-LLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓University of International Business and EconomicsBeijing · China7 works · 37 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju and Ming Zhang. 2025. A Survey on Efficient Large Language Model Training: From Data-centric Perspectives. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1493

@inproceedings{luo2025survey,
  title = {{A Survey on Efficient Large Language Model Training: From Data-centric Perspectives}},
  author = {Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju and Ming Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1493},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1493}
}

References

131 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul Hameed Azeemi, Ihsan Ayyub Qazi, and Agha Ali Raza. 2024. Language model-driven data pruning enables efficient active learning. arXiv preprint arXiv:2410.04275. find
  2. Randall Balestriero, Romain Cosentino, and Sarath Shekkizhar. 2024. Characterizing large language model geometry helps solve toxicity detection and generation. In Forty-first International Conference on Machine Learning. find
  3. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. arXiv preprint arXiv:2402.03927.
  4. Banooqa Banday, Kowshik Thopalli, Tanzima Z Islam, and Jayaraman J Thiagarajan. 2024. On the role of prompt construction in enhancing efficacy and efficiency of llm-based tabular data generation. arXiv preprint arXiv:2409.03946. — [arXiv:2409.03946](https://arxiv.org/abs/2409.03946) find
  5. Luca Becker, Philip Pracht, Peter Sertdal, Jil Uboreck, Alexander Bendel, and Rainer Martin. 2024. Conditional label smoothing for llm-based data augmentation in medical text classification. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 833–840. IEEE. find
  6. Gantavya Bhatt, Yifang Chen, Arnav M Das, Jifan Zhang, Sang T Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon S Du, Kevin Jamieson, et al. 2024. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint arXiv:2401.06692. find
  7. Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. 2024. Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.12030.
  8. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. find
  9. Xunxin Cai, Meng Xiao, Zhiyuan Ning, and Yuanchun Zhou. 2023. Resolving the imbalance issue in hierarchical disciplinary topic inference via llm-based data augmentation. In 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pages 1424–1429. IEEE. find
  10. Yash Prakash Chetnani. 2023. Evaluating the impact of model size on toxicity and stereotyping in generative llm. Master’s thesis, State University of New York at Buffalo. find
  11. Riccardo Cantini, Alessio Orsino, and Domenico Talia. 2024. Xai-driven knowledge distillation of large language models for efficient deployment on low-resource devices. Journal of Big Data, 11(1):63. find
  12. Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, and Kai Chen. 2025. Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement. arXiv preprint arXiv:2501.12273.
  13. Yung-Chieh Chan, George Pu, Apaar Shanker, Parth Suresh, Penn Jenks, John Heyer, and Sam Denton. 2024. Balancing cost and effectiveness of synthetic data generation strategies for llms. arXiv preprint arXiv:2409.19759. — [arXiv:2409.19759](https://arxiv.org/abs/2409.19759) find
  14. Kaiyan Chang, Kun Wang, Nan Yang, Ying Wang, Dantong Jin, Wenlong Zhu, Zhirong Chen, Cangyuan Li, Hao Yan, Yunhao Zhou, et al. 2024. Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework. In Proceedings of the 61st ACM/IEEE Design Automation Confer… find
  15. Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. 2024a. Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models. arXiv preprint arXiv:2406.12397. find
  16. Jiuhai Chen and Jonas Mueller. 2024. Automated data curation for robust language model fine-tuning. arXiv preprint arXiv:2403.12776. — [arXiv:2403.12776](https://arxiv.org/abs/2403.12776) find
  17. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701.
  18. Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024b. Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system. arXiv preprint arXiv:2410.08115. find
  19. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024c. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. find
  20. Juhwan Choi, Jungmin Yun, Kyohoon Jin, and Young-Bin Kim. 2024. Multi-news+: Cost-efficient dataset cleansing via llm-based data annotation. arXiv preprint arXiv:2404.09682. find
  21. Vikram S Chundawat, Ayush K Tarun, Murari Mandal, Mukund Lahoti, and Pratik Narang. 2022. A universal metric for robust evaluation of synthetic tabular data. IEEE Transactions on Artificial Intelligence, 5(1):300–309. find
  22. John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140.
  23. Yu Cui, Feng Liu, Pengbo Wang, Bohao Wang, Heng Tang, Yi Wan, Jun Wang, and Jiawei Chen. 2024. Distillation matters: empowering sequential recommenders to match the performance of large language models. In Proceedings of the 18th ACM Conference on Recommender Systems, pages 507–517. find
  24. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, et al. 2025. Auggpt: Leveraging chatgpt for text data augmentation. IEEE Transactions on Big Data.
  25. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335.
  26. Flavio Di Palo, Prateek Singhi, and Bilal Fadlallah. 2024. Performance-guided llm knowledge distillation for efficient text classification at scale. arXiv preprint arXiv:2411.05045. find
  27. Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. 2024. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.06961. — [arXiv:2410.06961](https://arxiv.org/abs/2410.06961) find
  28. Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. 2024. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631. find
  29. Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653. — [arXiv:2311.15653](https://arxiv.org/abs/2311.15653) find
  30. Zeyu Gan and Yong Liu. 2024. Towards a theoretical understanding of synthetic data in llm post-training: A reverse-bottleneck perspective. arXiv preprint arXiv:2410.01720. — [arXiv:2410.01720](https://arxiv.org/abs/2410.01720) find
  31. Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. 2024. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.… find
  32. Fabricio Goes, Zisen Zhou, Piotr Sawicki, Marek Grzes, and Daniel G Brown. 2022. Crowd score: A method for the evaluation of jokes using large language model ai voters as judges. arXiv preprint arXiv:2212.11214. — [arXiv:2212.11214](https://arxiv.org/abs/2212.11214) find
  33. Daya Guo, Dejian Yang, Haowei Zhang, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  34. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509.
  35. Chenghua Huang, Zhizhen Fan, Lu Wang, Fangkai Yang, Pu Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2024. Self-evolved reward learning for llms. arXiv preprint arXiv:2411.00418. — [arXiv:2411.00418](https://arxiv.org/abs/2411.00418) find
  36. Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, M Yang, and Mao Yang. 2023. Fewer is more: Boosting llm reasoning with reinforced context pruning. arXiv preprint arXiv:2312.08901. — [arXiv:2312.08901](https://arxiv.org/abs/2312.08901) find
  37. Zheng Hui, Zhaoxiao Guo, Hang Zhao, Juanyong Duan, and Congrui Huang. 2024. Toxicraft: A novel framework for synthetic generation of harmful information. arXiv preprint arXiv:2409.14740. find
  38. Daniel P Jeong, Zachary C Lipton, and Pradeep Ravikumar. 2024. Llm-select: Feature selection with large language models. arXiv preprint arXiv:2407.02694. — [arXiv:2407.02694](https://arxiv.org/abs/2407.02694) find
  39. Kaidi Jia, Yanxia Wu, and Rongsheng Li. 2024. Curriculum-style data augmentation for llm-based metaphor detection. arXiv preprint arXiv:2412.02956. — [arXiv:2412.02956](https://arxiv.org/abs/2412.02956) find
  40. Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2023. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing. arXiv preprint arXiv:2305.16635.
  41. Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. 2024. Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.02774. find
  42. S Karunya, M Jalakandeshwaran, Thanuja Babu, and R Uma. 2023. Ai-powered real-time speech-to-speech translation for virtual meetings using machine learning models. In 2023 Intelligent Computing and Control for Engineering and Business Systems (IC-CEBS), pages 1–6. IEEE. find
  43. Minsang Kim and Seungjun Baek. 2024. Measuring sample importance in data pruning for training llms from a data compression perspective. arXiv preprint arXiv:2406.14124. — [arXiv:2406.14124](https://arxiv.org/abs/2406.14124) find
  44. Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. 2023. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1813–1829. find
  45. Dong-Ho Lee, Hyundong Cho, Woojeong Jin, Jihyung Moon, Sungjoon Park, Paul Röttger, Jay Pujara, and Roy Ka-Wei Lee. 2024a. Improving covert toxicity detection by retrieving and generating references. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 266–274. find
  46. Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2024b. Llm2llm: Boosting llms with novel iterative data enhancement. arXiv preprint arXiv:2403.15042. find
  47. Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. 2024a. Calibraeval: Calibrating prediction distribution to mitigate selection bias in llms-as-judges. arXiv preprint arXiv:2410.15393.
  48. Lei Li, Yongfeng Zhang, and Li Chen. 2023a. Prompt distillation for efficient llm-based recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 1348–1357. find
  49. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023b. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032.
  50. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F Chen, Zhengyuan Liu, and Diyi Yang. 2023c. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638.
  51. Xinjin Li, Yu Ma, Yangchen Huang, Xingqi Wang, Yuzhen Lin, and Chenxi Zhang. 2024b. Synergized data efficiency and compression (sec) optimization for large language models. In 2024 4th International Conference on Electronic Information Engineering and Computer Science (EIECS), pages 586– 591. IEEE. find
  52. Yichuan Li, Kaize Ding, Jianling Wang, and Kyumin Lee. 2024c. Empowering large language models for textual data augmentation. arXiv preprint arXiv:2404.17642. find
  53. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126.
  54. Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. 2023d. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312.10302.
  55. Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations. find
  56. Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. 2024a. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407.14507. — [arXiv:2407.14507](https://arxiv.org/abs/2407.14507)
  57. Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. 2024b. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm. arXiv preprint arXiv:2408.08072. — [arXiv:2408.08072](https://arxiv.org/abs/2408.08072) find
  58. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  59. Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, and Jiaxuan You. 2024a. Arxiv copilot: A self-evolving and efficient llm system for personalized academic assistance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 122–130. find
  60. Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. 2024b. Data-efficient fine-tuning for llm-based recommendation. In Proceedings of the 47th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 365–374. find
  61. Philip Lippmann, Matthijs TJ Spaan, and Jie Yang. 2024. Illuminating blind spots of language models with targeted agent-in-the-loop synthetic data. arXiv preprint arXiv:2403.17860. — [arXiv:2403.17860](https://arxiv.org/abs/2403.17860) find
  62. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374.
  63. Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, et al. 2024a. Coachlm: Automatic instruction revisions improve the data quality in llm instruction tuning. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages… find
  64. Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. 2024b. Sample-efficient alignment for llms. arXiv preprint arXiv:2411.01493. — [arXiv:2411.01493](https://arxiv.org/abs/2411.01493) find
  65. Junyu Luo, Xiao Luo, Xiusi Chen, Zhiping Xiao, Wei Ju, and Ming Zhang. 2025a. Semi-supervised fine-tuning for large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2795–2808. find
  66. Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. 2024. Robustft: Robust supervised fine-tuning for large language models under noisy response. Preprint, arXiv:2412.14922. — [arXiv:2412.14922](https://arxiv.org/abs/2412.14922) find
  67. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025b. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460.
  68. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. find
  69. Tinh Son Luong, Thanh-Thien Le, Linh Ngo Van, and Thien Huu Nguyen. 2024. Realistic evaluation of toxicity in large language models. arXiv preprint arXiv:2405.10659. find
  70. Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, and Maria Lomeli. 2024. Source2synth: Synthetic data generation and curation grounded in real data sources. arXiv preprint arXiv:2409.08239. — [arXiv:2409.08239](https://arxiv.org/abs/2409.08239) find
  71. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  72. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36. find
  73. Daniel McDonald, Rachael Papadopoulos, and Leslie Benningfield. 2024. Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark. Authorea Preprints. find
  74. Xuran Ming, Shoubin Li, Mingyang Li, Lvlong He, and Qing Wang. 2024. Autolabel: Automated textual data annotation method based on active learning and large language model. In International Conference on Knowledge Science, Engineering and Management, pages 400–411. Springer. find
  75. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. In The Thirty-eighth Annual Conference on Neural In… find
  76. Ryumei Nakada, Yichen Xu, Lexin Li, and Linjun Zhang. 2024. Synthetic oversampling: Theory and a practical approach using llms to address data imbalance. arXiv preprint arXiv:2406.03628. — [arXiv:2406.03628](https://arxiv.org/abs/2406.03628) find
  77. Sejoon Oh, Yiqiao Jin, Megha Sharma, Donghyun Kim, Eric Ma, Gaurav Verma, and Srijan Kumar. 2024. Uniguard: Towards universal safety guardrails for jailbreak attacks on multimodal large language models. arXiv:2411.01703. — [arXiv:2411.01703](https://arxiv.org/abs/2411.01703) find
  78. Bo Pan, Zheng Zhang, Yifei Zhang, Yuntong Hu, and Liang Zhao. 2024a. Distilling large language models for text-attributed graph learning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1836–1845. find
  79. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023a. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188. find
  80. Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. 2023b. Reusing pre-trained models by multi-linear operators for efficient training. Advances in Neural Information Processing Systems, 36:3248–3262. find
  81. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. 2024b. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. arXiv preprint arXiv:2403.12968.
  82. Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. 2024. Improving data efficiency via curating llm-driven rating systems. arXiv preprint arXiv:2410.10877. — [arXiv:2410.10877](https://arxiv.org/abs/2410.10877) find
  83. Nicholas Pangakis and Samuel Wolken. 2024. Knowledge distillation in automated annotation: Supervised text classification with llm-generated training labels. arXiv preprint arXiv:2406.17633.
  84. Changhua Pei, Zihan Liu, Jianhui Li, Erhan Zhang, Le Zhang, Haiming Zhang, Wei Chen, Dan Pei, and Gaogang Xie. 2024. Self-evolutionary group-wise log parsing based on large language model. In 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), pages 49–60. IEEE. find
  85. Zheyan Qu, Lu Yin, Zitong Yu, Wenbo Wang, et al. 2024. Coursegpt-zh: an educational large language model based on knowledge distillation incorporating prompt optimization. arXiv preprint arXiv:2405.04781. — [arXiv:2405.04781](https://arxiv.org/abs/2405.04781) find
  86. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
  87. Indira Sen, Dennis Assenmacher, Mattia Samory, Isabelle Augenstein, Wil van der Aalst, and Claudia Wagner. 2023. People make better edits: Measuring the efficacy of llm-generated counterfactually augmented data for harmful language detection. arXiv preprint arXiv:2311.01270. find
  88. Minju Seo, Jinheon Baek, James Thorne, and Sung Ju Hwang. 2024. Retrieval-augmented data augmentation for low-resource domain tasks. arXiv preprint arXiv:2402.13482. — [arXiv:2402.13482](https://arxiv.org/abs/2402.13482) find
  89. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin…
  90. Yi Tang, Chia-Ming Chang, and Xi Yang. 2024. Pdfchatannotator: A human-llm collaborative multimodal data annotation tool for pdf-format catalogs. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 419–430. find
  91. Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. 2024. A survey on self-evolution of large language models. arXiv preprint arXiv:2404.14387. — [arXiv:2404.14387](https://arxiv.org/abs/2404.14387) find
  92. Vithursan Thangarasa, Ganesh Venkatesh, Mike Lasby, Nish Sinnadurai, and Sean Lie. 2024. Self-data distillation for recovering quality in pruned large language models. arXiv preprint arXiv:2410.09982. find
  93. Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. 2023. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863.
  94. Fei Wang, Ninareh Mehrabi, Palash Goyal, Rahul Gupta, Kai-Wei Chang, and Aram Galstyan. 2024a. Data advisor: Dynamic data curation for safety alignment of large language models. arXiv preprint arXiv:2410.05269. find
  95. Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. 2024b. A survey on data selection for llm instruction tuning. arXiv preprint arXiv:2402.05123. — [arXiv:2402.05123](https://arxiv.org/abs/2402.05123) find
  96. Peidong Wang, Ming Wang, Zhiming Ma, Xiaocui Yang, Shi Feng, Daling Wang, and Yifei Zhang. 2024c. Language models as continuous self-evolving data engineers. arXiv preprint arXiv:2412.15151. find
  97. Pengkun Wang, Zhe Zhao, HaiBin Wen, Fanfu Wang, Binwu Wang, Qingfu Zhang, and Yang Wang. 2024d. Llm-autoda: Large language model-driven automatic data augmentation for long-tailed problems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  98. Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. 2024e. Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. arXiv preprint arXiv:2402.11443.
  99. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024f. Self-taught evaluators. arXiv preprint arXiv:2408.02666.
  100. Yau-Shian Wang and Yingshan Chang. 2022. Toxicity detection with generative prompt-based inference. arXiv preprint arXiv:2205.12390. — [arXiv:2205.12390](https://arxiv.org/abs/2205.12390) find
  101. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 13484–1350… find
  102. Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. 2024a. Interactive evolution: A neural-symbolic self-training framework for large language models. arXiv preprint arXiv:2406.11736.
  103. Shengzhe Xu, Cho-Ting Lee, Mandar Sharma, Raquib Bin Yousuf, Nikhil Muralidhar, and Naren Ramakrishnan. 2024b. Are llms naturally good at synthetic tabular data generation? arXiv preprint arXiv:2406.14541. — [arXiv:2406.14541](https://arxiv.org/abs/2406.14541) find
  104. Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. 2023. Rethinking the instruction quality: Lift is what you need. CoRR. find
  105. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024c. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464.
  106. Shuangtao Yang, Xiaoyi Liu, Xiaozheng Dong, and Bo Fu. 2024a. Mini-da: Improving your model performance through minimal data augmentation using llm. In Proceedings of the Fifth Workshop on Data Science with Human-in-the-Loop (DaSH 2024), pages 25–30. find
  107. Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, et al. 2024g. Memoryllm: Towards self-updatable large language models. arXiv preprint arXiv:2402.04624.
  108. Yingxuan Yang, Huayi Wang, Muning Wen, Xiaoyun Mo, Qiuying Peng, Jun Wang, and Weinan Zhang. 2024b. P3: A policy-driven, pace-adaptive, and diversity-promoted framework for data pruning in llm training. arXiv preprint arXiv:2408.05541. — [arXiv:2408.05541](https://arxiv.org/abs/2408.05541) find
  109. Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2024h. Harmonic: Harnessing llms for tabular data synthesis and privacy protection. arXiv preprint arXiv:2408.02927. — [arXiv:2408.02927](https://arxiv.org/abs/2408.02927) find
  110. Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568. — [arXiv:2402.14568](https://arxiv.org/abs/2402.14568) find
  111. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594.
  112. Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. arXiv preprint arXiv:2305.14327. find
  113. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333.
  114. Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. 2024. Chatts: Aligning time series with llms via synthetic data for enhanced understanding and reasoning. arXiv preprint arXiv:2412.03104. — [arXiv:2412.03104](https://arxiv.org/abs/2412.03104) find
  115. Huajian Xin, Daya Guo, Zhihong Shao, ZZ Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Advancing theorem proving in llms through Junjie Oscar Yin and Alexander M Rush. 2024. Compute-constrained data selection. arXiv preprint arXiv:2410.16208. — [arXiv:2410.16208](https://arxiv.org/abs/2410.16208) find
  116. Jiaxuan You, Mingjie Liu, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Llm-evolve: Evaluation for llm’s evolving capability on benchmarks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16937–16942. find
  117. Simon Yu, Liangyu Chen, Sara Ahmadian, and Marzieh Fadaee. 2024. Diversify and conquer: Diversity-centric data selection with iterative refinement. arXiv preprint arXiv:2409.11378. — [arXiv:2409.11378](https://arxiv.org/abs/2409.11378) find
  118. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024b. Self-rewarding language models. arXiv preprint arXiv:2401.10020. find
  119. Ye Yuan, Chengwu Liu, Jingyang Yuan, Gongbo Sun, Siqi Li, and Ming Zhang. 2024c. A hybrid rag system with comprehensive enhancement on complex reasoning. arXiv preprint arXiv:2408.05141. — [arXiv:2408.05141](https://arxiv.org/abs/2408.05141) find
  120. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488.
  121. Oleg Zendel, J Shane Culpepper, Falk Scholer, and Paul Thomas. 2024. Enhancing human annotation: Leveraging large language models and efficient batch processing. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, pages 340–345. find
  122. Jiang Zhang, Qiong Wu, Yiming Xu, Cheng Cao, Zheng Du, and Konstantinos Psounis. 2024a. Efficient toxic content detection by bootstrapping and distilling large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21779–21787. find
  123. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023a. Llmaaa: Making large language models as active annotators. arXiv preprint arXiv:2310.19596. find
  124. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023b. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
  125. Yuzhe Zhang, Huan Liu, Yang Xiao, Mohammed Amoon, Dalin Zhang, Di Wang, Shusen Yang, and Chai Quek. 2024b. Llm-enhanced multi-teacher knowledge distillation for modality-incomplete emotion recognition in daily healthcare. IEEE Journal of Biomedical and Health Informatics. find
  126. Ziyan Zhang, Yang Hou, Chen Gong, and Zhenghua Li. 2025. Data augmentation for cross-domain parsing via lightweight llm generation and tree hybridization. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11235–11247. find
  127. Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay Yoon Lee, Mohit Iyyer, and Andrew McCallum. 2024a. Multistage collaborative knowledge distillation from a large language model for semi-supervised sequence generation. In Proceedings of the 62nd Annual Meeting of t… find
  128. Wanru Zhao, Hongxiang Fan, Shell Xu Hu, Wangchunshu Zhou, and Nicholas Donald Lane. 2024b. CLUES: Collaborative private-domain high-quality data selection for llms via training dynamics. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS). find
  129. Yang Zhou, Shimin Shan, Hongkui Wei, Zhehuan Zhao, and Wenshuo Feng. 2024a. Pga-scire: Harnessing llm on data augmentation for enhancing scientific relation extraction. arXiv preprint arXiv:2405.20787. — [arXiv:2405.20787](https://arxiv.org/abs/2405.20787) find
  130. Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, and Yuan Wu. 2024b. A survey on data augmentation in large model era. arXiv preprint arXiv:2401.15422. — [arXiv:2401.15422](https://arxiv.org/abs/2401.15422) find
  131. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2025-survey-efficient-large-language-model/