Elamur Research⌕
☰
home / papers / 2025 / Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models

Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models

Paper · 2025 · Lingyuan Liu and Mengxiang Zhang

Summary

White-box knowledge distillation (KD) for LLMs has mostly been improved by designing new divergence losses or by mixing ground-truth and student-generated responses. The authors argue that two factors get too little attention: how good the training data is and how well it suits the student. Generating student outputs on-policy is also costly. They propose Selective Reflection Distillation (SRD), a plug-and-play data-curation layer that leaves the KD algorithm unchanged. The student first answers each training prompt. Difficulty is then estimated from two signals, ROUGE-L against the reference and cross-entropy loss, which are combined through reciprocal rank fusion. The hardest fraction of examples is removed. The remaining data is divided into difficulty tiers and fed in from easy to hard at fixed intervals, while the temperature rises and the supervised-loss weight falls over training. Across several model families, tasks and KD losses, SRD trains on less data, cuts runtime by up to 39%, and improves student quality. This makes data-centric choices a practical lever for compressing LLMs.

Key points

  • Settings tested: Instruction following, math reasoning, code generation, summarization and machine translation, using GPT-2, OpenLLaMA2, Qwen2.5 and T5. SRD was combined with off-policy losses (KLD, RKL, JSD, TVD, SKL, SRKL) and with on-policy GKD.
  • OpenLLaMA2-7B → OpenLLaMA2-3B: SRD increased ROUGE-L for every baseline across five instruction-following benchmarks. It used about 25% less training data and reduced runtime by roughly 35%.
  • SRD vs. traditional curriculum learning (TCL), with KLD:
    • OpenLLaMA2: average ROUGE-L rose from 22.56 to 26.07 with SRD (+15.53%), compared with 22.91 for TCL.
    • GPT-2 1.5B→0.1B: average ROUGE-L rose from 18.10 to 18.96 with SRD, while TCL fell to 17.91.
  • Hyperparameters:
    • The retention ratio λ shows diminishing returns, suggesting the hardest samples add little.
    • A moderate number of curriculum stages works best.
    • The final temperature and the initial SFT ratio each have an optimal middle range.
  • Code: Available at github.com/liuliuyuan6/SRD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Lingyuan Liu and Mengxiang Zhang. 2025. Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models. arXiv:2508.06135.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.06135 · arXiv 2508.06135

@misc{liu2025less,
  title = {{Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models}},
  author = {Lingyuan Liu and Mengxiang Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2508.06135},
  eprint = {2508.06135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.06135}
}

References

58 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Garea, S. R.; Geist, M.; and Bachem, O. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations.
  2. Anthropic. 2024. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Published: 21 Jun 2024. ↗
  3. Aryan, A.; Nain, A. K.; McMahon, A.; Meyer, L. A.; and Sahota, H. S. 2023. The costly dilemma: generalization, evaluation and cost-optimal deployment of large language models. arXiv preprint arXiv:2308.08061. — [arXiv:2308.08061](https://arxiv.org/abs/2308.08061) find
  4. Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  5. Bengio, Y.; Louradour, J.; Collobert, R.; and Weston, J. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, 41–48.
  6. Cettolo, M.; Federico, M.; Bentivogli, L.; Niehues, J.; Stüker, S.; Sudoh, K.; Yoshino, K.; and Federmann, C. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, 2–14.
  7. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2(3): 6. ↗
  8. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  9. Conover, M.; Hayes, M.; Mathur, A.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. find
  10. Cormack, G. V.; Clarke, C. L.; and Buettcher, S. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 758–759.
  11. Ding, N.; Chen, Y.; Xu, B.; Qin, Y.; Zheng, Z.; Hu, S.; Liu, Z.; Sun, M.; and Zhou, B. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.
  12. Gao, S.; Shi, Z.; Zhu, M.; Fang, B.; Xin, X.; Ren, P.; Chen, Z.; Ma, J.; and Ren, Z. 2024. Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18030–18038. find
  13. Gliwa, B.; Mochol, I.; Biesek, M.; and Wawer, A. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237.
  14. Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2023. MiniLLM: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543.
  15. Gunasekar, S.; Zhang, Y.; Aneja, J.; Mendes, C. C. T.; Del Giorno, A.; Gopi, S.; Javaheripi, M.; Kauffmann, P.; de Rosa, G.; Saarikivi, O.; et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644. find
  16. Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  17. Honovich, O.; Scialom, T.; Levy, O.; and Schick, T. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689.
  18. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2): 3.
  19. Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  20. Kim, Y.; and Rush, A. M. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, 1317–1327.
  21. Ko, J.; Chen, T.; Kim, S.; Ding, T.; Liang, L.; Zharkov, I.; and Yun, S.-Y. 2025. DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs. arXiv preprint arXiv:2503.07067.
  22. Ko, J.; Kim, S.; Chen, T.; and Yun, S.-Y. 2024. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898.
  23. Li, J.; Tang, T.; Zhao, W. X.; Nie, J.-Y.; and Wen, J.-R. 2024a. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9): 1–39. find
  24. Li, M.; Chen, L.; Chen, J.; He, S.; Gu, J.; and Zhou, T. 2024b. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024, 16189–16211. find
  25. Li, M.; Zhang, Y.; Li, Z.; Chen, J.; Chen, L.; Cheng, N.; Wang, J.; Zhou, T.; and Xiao, J. 2023a. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032.
  26. Li, Z.; Li, X.; Yang, L.; Zhao, B.; Song, R.; Luo, L.; Li, J.; and Yang, J. 2023b. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 1504–1512. find
  27. Libby, M. E.; Weiss, J. S.; Bancroft, S.; and Ahearn, W. H. 2008. A comparison of most-to-least and least-to-most prompting on the acquisition of solitary play skills. Behavior analysis in practice, 1: 37–43. find
  28. Lin, A.; Wohlwend, J.; Chen, H.; and Lei, T. 2020. Autoregressive knowledge distillation through imitation learning. arXiv preprint arXiv:2009.07253. find
  29. Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81.
  30. Liu, J.; Xia, C. S.; Wang, Y.; and Zhang, L. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36: 21558–21572.
  31. Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  32. Liu, L.; and Zhang, M. 2025. Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework. arXiv preprint arXiv:2506.05695. find
  33. Luo, Z.; Xu, C.; Zhao, P.; Sun, Q.; Geng, X.; Hu, W.; Tao, C.; Ma, J.; Lin, Q.; and Jiang, D. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. find
  34. Maslej, N.; Fattorini, L.; Perrault, R.; Gil, Y.; Parli, V.; Kariuki, N.; Capstick, E.; Reuel, A.; Brynjolfsson, E.; Etchemendy, J.; Ligett, K.; Lyons, T.; Manyika, J.; Niebles, J. C.; Shoham, Y.; Wald, R.; Walsh, T.; Hamrah, A.; Santarlasci, L.; Lotufo, J. B.; Rome, A.; Shi, A.; and Oak, S. 2025. find
  35. The AI Index 2025 Annual Report. Technical report, Stanford University, Institute for Human-Centered Artificial Intelligence (HAI), Stanford, CA. AI Index Steering Committee. find
  36. Narayan, S.; Cohen, S. B.; and Lapata, M. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
  37. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744.
  38. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
  39. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9. find
  40. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1–67.
  41. Sengupta, A.; Dixit, S.; Akhtar, M. S.; and Chakraborty, T. 2023. A good learner can teach better: Teacher-student collaborative knowledge distillation. In Proceedings of the The Twelfth International Conference on Learning Representations, Virtual Event, 25–29. find
  42. Team, Q. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2407.10671. find
  43. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  44. Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022a. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. find
  45. Wang, G.; Yang, Z.; Wang, Z.; Wang, S.; Xu, Q.; and Huang, Q. 2025. ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via alpha-beta-Divergence. arXiv preprint arXiv:2505.04560. — [arXiv:2505.04560](https://arxiv.org/abs/2505.04560) find
  46. Wang, Y.; Mishra, S.; Alipoormolabashi, P.; Kordi, Y.; Mirzaei, A.; Arunkumar, A.; Ashok, A.; Dhanasekaran, A. S.; Naik, A.; Stap, D.; et al. 2022b. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2: 2. find
  47. Wang, X.; Chen, Y.; and Zhu, W. 2021. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9): 4555–4576. find
  48. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824–24837.
  49. Wen, Y.; Li, Z.; Du, W.; and Mou, L. 2023. F-divergence minimization for sequence-level knowledge distillation. arXiv preprint arXiv:2307.15190.
  50. Wu, T.; Tao, C.; Wang, J.; Yang, R.; Zhao, Z.; and Wong, N. 2024. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657.
  51. Xiang, L.; Ding, G.; and Han, J. 2020. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, 247–263. find
  52. Springer. Xu, C.; Sun, Q.; Zheng, K.; Geng, X.; Zhao, P.; Feng, J.; Tao, C.; Lin, Q.; and Jiang, D. 2024a. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
  53. Xu, X.; Li, M.; Tao, C.; Shen, T.; Cheng, R.; Li, J.; Xu, C.; Tao, D.; and Zhou, T. 2024b. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  54. Xue, L.; Constant, N.; Roberts, A.; Kale, M.; Al-Rfou, R.; Siddhant, A.; Barua, A.; and Raffel, C. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
  55. Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  56. Yang, C.; Zhu, Y.; Lu, W.; Wang, Y.; Chen, Q.; Gao, C.; Yan, B.; and Chen, Y. 2024b. Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology. find
  57. Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  58. Zhu, Q.; Chen, X.; Wu, P.; Liu, J.; and Zhao, D. 2021. Combining curriculum learning and knowledge distillation for dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, 1284–1295. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-less-more-selective-reflection-compatible/