Elamur Research⌕
☰
home / papers / 2025 / LLM-Oriented Token-Adaptive Knowledge Distillation

LLM-Oriented Token-Adaptive Knowledge Distillation

Paper · 2025 · Xurong Xie, Zhucun Xue, Jiafu Wu et al.

Summary

Logit-based knowledge distillation is a common way to compress large language models. Standard recipes, however, weight every target token equally and use one global temperature, which ignores how the student's mastery of individual tokens changes during training. The authors (Zhejiang University and Tencent Youtu Lab) propose AdaKD, a plug-in framework that scores each token's difficulty using the Hellinger distance between the teacher's and the student's output distributions. That single score drives two components:

  • LATF follows a smoothed training loss and widens or narrows the fraction of hardest tokens that receive distillation loss, discarding tokens the student has already learned.
  • IDTS assigns each kept token its own temperature, inversely tied to difficulty. Hard tokens get sharp, low temperatures for targeted correction, while easy tokens get high temperatures so the student absorbs the teacher's full, smooth distribution.

On instruction-following benchmarks, AdaKD consistently improved several existing distillation objectives across multiple model architectures, with only a small throughput cost. It offers a cheap, objective-agnostic upgrade for LLM compression.

Key points

  • A token-level analysis motivated the design. Gradients from "easy" tokens were tiny, nearly orthogonal to the batch gradient, and sometimes opposed the SFT direction.
  • The difficulty metric is the Hellinger distance, bounded in [0, 1]. It is symmetric and sensitive to disagreements on low-probability candidates.
  • The chosen LATF settings were tolerance ϵ = 0.05, step δ = 0.05, EMA decay β = 0.97, and warm-up ratio 0.05. On the UnNI sensitivity grid, performance was stable when δ ≤ ϵ, with a top score of 43.23. For IDTS, a modulation intensity of c = 0.5 gave the best average ROUGE-L.
  • The overhead is small. On Qwen2, training throughput fell by 3.29% with RKD, 0.87% with GKD, and 1.55% with DistiLLM. IDTS also outperformed fixed temperatures, CTKD, and Logit Std.
  • The authors list several limitations. Evaluation covers instruction-following only, LATF's discrete updates can cause slight oscillations, and IDTS is heuristic. The code is available at github.com/SassyRong/AdaKD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China134 works · 19,743 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu and Jiangning Zhang. 2025. LLM-Oriented Token-Adaptive Knowledge Distillation. arXiv:2510.11615.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.11615 · arXiv 2510.11615

@misc{xie2025llm,
  title = {{LLM-Oriented Token-Adaptive Knowledge Distillation}},
  author = {Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu and Jiangning Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2510.11615},
  eprint = {2510.11615},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.11615}
}

References

46 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  3. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  4. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  5. Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023.
  6. Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. A review on edge large language models: Design, execution, and applications. ACM Computing Surveys, 57(8):1–35, 2025. find
  7. Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. Beyond efficiency: A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625, 2024. — [arXiv:2401.00625](https://arxiv.org/abs/2401.00625) find
  8. Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. arXiv:2407.06023, 2024. Distilling system 2 into system 1. arXiv preprint find
  9. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023.
  10. Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022.
  11. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  12. Anthropic. Introducing Claude. https://www.anthropic.com/news/introducing-claude, 2023. ↗
  13. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019.
  14. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020. find
  15. Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, 2023. find
  16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  17. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In ICLR, 2024. find
  18. Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657, 2024.
  19. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning, 2024.
  20. Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation via α-β-divergence. arXiv preprint arXiv:2505.04560, 2025. — [arXiv:2505.04560](https://arxiv.org/abs/2505.04560) find
  21. Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. PBR, 2014. find
  22. Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. Selective knowledge distillation for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  23. Qinhong Zhou, Peng Li, Yang Liu, Yuyang Guan, Qizhou Xing, Ming Chen, and Maosong Sun. Adads: Adaptive data selection for accelerating pre-trained language model knowledge distillation. AI Open, 4:56–63, 2023. find
  24. Songming Zhang, Yunlong Liang, Shuaibo Wang, Wenjuan Han, Jian Liu, Jinan Xu, and Yufeng Chen. Towards understanding and improving knowledge distillation for neural machine translation. arXiv preprint arXiv:2305.08096, 2023. find
  25. Tianyu Peng and Jiajun Zhang. Enhancing knowledge distillation of large language models through efficient multi-modal distribution alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2478–2496, 2025.
  26. Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. find
  27. Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong, and Yunhe Wang. Multi-granularity semantic revision for large language model distillation. arXiv preprint arXiv:2407.10068, 2024.
  28. Jia Guo. Reducing the teacher-student gap via adaptive temperatures. arXiv preprint arXiv:2010.07485, 2020. — [arXiv:2010.07485](https://arxiv.org/abs/2010.07485) find
  29. Zhihao Chi, Tu Zheng, Hengjia Li, Zheng Yang, Boxi Wu, Binbin Lin, and Deng Cai. Normkd: Normalized logits for knowledge distillation. arXiv preprint arXiv:2308.00520, 2023. — [arXiv:2308.00520](https://arxiv.org/abs/2308.00520) find
  30. Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504–1512, 2023. find
  31. Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. arXiv preprint arXiv:2104.07163, 2021. find
  32. Shunzhi Yang, Xiong Yang, Jin Ren, Liuchi Xu, Jinfeng Yang, Zhenhua Huang, Zheng Gong, and Wenguang Wang. Adaptive temperature distillation method for mining hard samples’ knowledge. Neurocomputing, 636: 129745, 2025. find
  33. Jun Long, Zhuoying Yin, Yan Han, and Wenti Huang. Mkdat: Multi-level knowledge distillation with adaptive temperature for distantly supervised relation extraction. Information, 15(7):382, 2024. find
  34. Ernst Hellinger. Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik, 1909(136):210–271, 1909. find
  35. Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. find
  36. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instructiontuned llm. 2023. find
  37. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
  38. Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2(3):6, 2023. ↗
  39. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705,… find
  40. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022.
  41. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024.
  42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… find
  43. Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, 2023. find
  44. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  45. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  46. Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15731–15740, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2025-llm-oriented-token-adaptive-knowledge/