Elamur Research⌕
☰
home / papers / 2026 / TIP: Token Importance in On-Policy Distillation

TIP: Token Importance in On-Policy Distillation

Paper · 2026 · Yuanda Xu, Hejian Sang, Zhengze Zhou et al.

Summary

On-policy distillation (OPD) trains a student model on sequences it generates itself, with the teacher giving feedback at every token. Usually all positions count equally. This paper asks which positions actually carry useful signal. The authors propose TIP, a taxonomy that places each token on two axes OPD already computes: normalized student entropy and student–teacher KL divergence. Tokens where the student is uncertain are easy to find. A second, smaller group is harder to spot: tokens where the student is confident but the teacher strongly disagrees. The authors prove that any rule based only on entropy cannot see these overconfident tokens. To capture both groups, they introduce a parameter-free Soft-OR score. The evaluation uses three teacher–student pairs across MATH-500, AIME 2024/2025, and the agentic DeepPlanning benchmark. Keeping only half of the tokens matches or beats full-token training while sharply lowering peak memory. Soft-OR generally outperforms entropy-based selection on math. The work offers a cheaper, more targeted way to distill reasoning models under limited GPU budgets.

Key points

  • Taxonomy: TIP defines four quadrants: Q1 (uncertain, teacher disagrees), Q2 (uncertain, teacher agrees), Q3 (overconfident) and Q4 (solved). Q4 makes up roughly 40–47% of tokens, while Q3 is only 3–15% but contains dense corrective signal.
  • Memory savings: Entropy-based selection that keeps 50% of tokens matches or exceeds all-token OPD and reduces peak training memory by up to 47%. More aggressive pruning saves up to 58%.
  • Math results: Models tested were Qwen3-8B→4B, Llama-70B→8B and Qwen2.5-14B→1.5B. Training on under 10% of tokens, selected for overconfidence, nearly matches full training. For Qwen3, Soft-OR at 50% raises MATH-500 from 76.7 to 79.1.
  • DeepPlanning results: With a Qwen3-1.7B student, Q3-only training on 20% of tokens beats full OPD (12.6 vs. 11.7 with a 14B teacher; 13.6 vs. 12.8 with a 32B teacher). Teacher entropy proved uninformative as a selection signal.
  • Code: The implementation extends the open-source repository HJSang/OPSD_OnPolicyDistillation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 328 works

Cite

Cite as Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang and Alborz Geramifard. 2026. TIP: Token Importance in On-Policy Distillation. arXiv:2604.14084.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.14084 · arXiv 2604.14084

@misc{xu2026tip,
  title = {{TIP: Token Importance in On-Policy Distillation}},
  author = {Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang and Alborz Geramifard},
  year = {2026},
  doi = {10.48550/arxiv.2604.14084},
  eprint = {2604.14084},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.14084}
}

References

36 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2306.13… ↗
  2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. ICML, 2009.
  3. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint ar…
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models. 2024. URL https://arxiv.org/abs/2407.21783. ↗
  5. Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. MiniPLM: Knowledge distillation for pre-training language models. arXiv preprint arXiv:2410.17215, 2024.
  6. Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. MiniPLM: Knowledge distillation for pre-training language models. 2025. URL https://openreview.net/forum? id=tJHDw8XfeC. ↗
  7. Yiju Guo, Wenkai Yang, Zexu Sun, Ning Ding, Zhiyuan Liu, and Yankai Lin. Learning to focus: Causal attention distillation via gradient-guided token pruning. In Advances in Neural Information Processing Systems, volume 38, 2025. find
  8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021.
  9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  10. Haiduo Huang, Jiangcheng Song, Yadong Zhang, and Pengju Ren. SelecTKD: Selective token-weighted knowledge distillation for LLMs. arXiv preprint arXiv:2510.24021, 2025. — [arXiv:2510.24021](https://arxiv.org/abs/2510.24021) find
  11. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://arxiv.org/abs/2305.12870. ↗
  12. Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026.
  13. Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning (ICML), 2018. find
  14. Minsang Kim and Seung Jun Baek. Explain in your own words: Improving reasoning via token-selective dual knowledge distillation. arXiv preprint arXiv:2603.13260, 2026.
  15. Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.07947. ↗
  16. M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. NeurIPS, 2010. find
  17. Zhen Peng et al. Adaswitch: Adaptive switching between teacher and student for on-policy distillation. arXiv preprint, 2025. find
  18. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  19. Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), 2018. find
  20. Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. CRISP: Compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433, 2026.
  21. Almog Tavor, Itay Ebenspanger, Neil Cnaan, and Mor Geva. Rethinking selective knowledge distillation. arXiv preprint arXiv:2602.01395, 2026. — [arXiv:2602.01395](https://arxiv.org/abs/2602.01395) find
  22. Jiapeng Wang, Yiwen Hu, Yanzipeng Gao, Haoyu Wang, Shuo Wang, Hongyu Lu, Jiaxin Mao, Wayne Xin Zhao, Junyi Li, and Ji-Rong Wen. Entropy-guided token dropout: Training autoregressive language models with limited domain data. arXiv preprint arXiv:2512.23422, 2025a. — [arXiv:2512.23422](https://arxiv.org/abs/2512.23422) find
  23. Shaobo Wang, Jiaming Wang, Jiajun Zhang, Cong Wang, Yue Min, Zichen Wen, Xingzhang Ren, Fei Huang, Huiqiang Jiang, Junyang Lin, Dayiheng Liu, and Linfeng Zhang. Winning the pruning gamble: A unified approach to joint sample and token pruning for efficient supervised fine-tuning. arXiv preprint arXiv… — [arXiv:2509.23873](https://arxiv.org/abs/2509.23873) find
  24. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement le… — Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
  25. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.org/abs/2002. 10957. ↗
  26. Jianghao Wu, Yasmeen George, Jin Ye, Yicheng Wu, Daniel F. Schmidt, and Jianfei Cai. SPINE: Token-selective test-time reinforcement learning with entropy-band regularization. arXiv preprint arXiv:2511.17938, 2025. — [arXiv:2511.17938](https://arxiv.org/abs/2511.17938) find
  27. Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, and Maosong Sun. Densing law of LLMs. Nature Machine Intelligence, 2025.
  28. Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token-adaptive knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34070–34078, 2026.
  29. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. Overconfident errors need stronger correction: Asymmetric confidence penalties for reinforcement learning. arXiv preprint arXiv:2602.21420, 2026a. — [arXiv:2602.21420](https://arxiv.org/abs/2602.21420) find
  30. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. P ACED : Distillation and self-distillation at the frontier of student competence. arXiv preprint, 2026b. find
  31. Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. OneBit: Towards extremely low-bit large language models. In Advances in Neural Information Processing Systems, volume 37, 2024. find
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  33. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli… — DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  34. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In Advances in Neural Information Processing Systems, volume 38, 2025. Oral.
  35. Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv, and Junyang Lin. DeepPlanning: Benchmarking long-horizon agentic planning with verifiable constraints. arXiv preprint arXiv:2601.18137, 2026.
  36. Chenghua Zhu, Siyan Wu, Xiangkang Zeng, Zishan Xu, Zhaolu Kang, Yifu Guo, Yuquan Lu, Junduan Huang, Guojing Zhou, et al. EDIS: Diagnosing LLM reasoning via entropy dynamics. arXiv preprint arXiv:2602.01288, 2026. — [arXiv:2602.01288](https://arxiv.org/abs/2602.01288) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-tip-token-importance-policy-distillation/