Elamur Research⌕
☰
home / papers / 2026 / Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation

Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation

Paper · 2026 · Lechen Zhang, Yunxiang Zhang, Wei Hu et al.

Summary

Distilling reasoning from large models such as DeepSeek-R1 into smaller LLMs usually relies on large SFT datasets in which every example counts the same. That approach ignores both which skills a problem requires and what the student model already knows. The authors propose a skill-centric distillation framework for math reasoning built on a hierarchical skill tree taken from Instruct-SkillMix. First, Qwen2.5-32B-Instruct labels each problem in a 100K OpenMathReasoning pool by walking the tree from the root down to the relevant leaf skills. Second, the student is evaluated on the pool, and training examples are drawn with probability inversely tied to the student's accuracy on each skill. Third, the full skill chain is inserted before each solution, so the model learns to name the skills it needs before it starts reasoning. With only 1K selected examples, this beats random selection on Qwen3-4B and Qwen3-8B. The gains concentrate on weaker skills, which makes per-skill performance more even. The work shows that careful, model-adaptive data selection can count for more than data volume.

Key points

  • Gains over random 1K SFT: +1.6 Avg@8 on Qwen3-4B (68.4 vs. 66.8) and +1.4 on Qwen3-8B (69.5 vs. 68.1). Evaluation used five benchmarks: AMC23, AIME2024, AIME2025, MATH L5 and OlympiadBench.
  • Full-corpus fine-tuning hurts: training on all 100K examples drops accuracy below the base model (58.1 vs. 67.1 for Qwen3-4B).
  • Sampling ablation: an accuracy exponent of T = 1.0 works well, and more aggressive settings saturate.
  • Skill-chain ablation: the full chain (72.9) beats root-only (72.2) and leaf-only (72.7) skill labels.
  • Generalization (Appendix C): the method also works with the EvalTree taxonomy and with R1-Distill-Llama-8B. It outperforms the LIMO, s1, Light-R1 and Select2Reason baselines. Code and data are released at github.com/orange0629/skill-data-selection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksEfficient models & compression · 297 works

Cite

Cite as Lechen Zhang, Yunxiang Zhang, Wei Hu and Lu Wang. 2026. Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.49

@inproceedings{zhang2026skill,
  title = {{Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation}},
  author = {Lechen Zhang, Yunxiang Zhang, Wei Hu and Lu Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.49},
  url = {https://doi.org/10.18653/v1/2026.acl-short.49}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Re. 2023. Skill-it! a data-driven skills framework for understanding and training language models. In Thirty-seventh Conference on Neural Information Processing Systems. find
  2. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llm…
  3. Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. 2024. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving. In The Thirty-eighth Annual… find
  4. Robert Gagne. 1962. The acquisition of knowledge. Psychological Review, 69:355–365. find
  5. Shousheng Jia Haosheng Zou, Xiaowei Lv and Xiangzheng Zhang. 2024. 360-llama-factory. find
  6. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proce…
  7. Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. 2026. STAT: Skill-targeted adaptive training. In The Fourteenth International Conference on Learning Representations. find
  8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
  9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.
  10. Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. 2025. Instruct-skillmix: A powerful pipeline for LLM instruction tuning. In The Thirteenth International Conference on Learning Representations. find
  11. Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, JUN ZHOU, Yanfang Ye, and Chuxu Zhang. 2025a. MASS: Mathematical data selection via skill graphs for pretraining large language models. In Forty-second International Conference on Machine Learning. find
  12. Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason Weston, Shang-Wen Li, Dong Wang, Ilia Kulikov, and Xian Li. 2025b. Naturalthoughts: Selecting and distilling reasoning traces for general reasoning tasks. Preprint, arXiv:2507.01921. — [arXiv:2507.01921](https://arxiv.org/abs/2507.01921) find
  13. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Rev…
  14. Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. 2025. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891. find
  15. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393.
  16. Shousheng Jia, and Xiangzheng Zhang. 2025. Lightr1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 318– 327, Vienna, Austria. Association for Computational L… find
  17. Richard T. White. 1973. Research into learning hierarchies. Review of Educational Research, 43(3):361– 375. find
  18. Cehao Yang, Xueyuan Lin, Xiaojun Wu, Chengjin Xu, Xuhui Jiang, Honghao Liu, Hui Xiong, and Jian Guo. 2025a. Select2reason: Efficient instruction-tuning data selection for long-cot reasoning. Preprint, arXiv:2505.17266.
  19. Zheyuan Yang, Lyuhao Chen, Arman Cohan, and Yilun Zhao. 2025b. Table-r1: Inference-time scaling for table reasoning. Preprint, arXiv:2505.23621. — [arXiv:2505.23621](https://arxiv.org/abs/2505.23621) find
  20. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. LIMO: Less is more for reasoning. In Second Conference on Language Modeling.
  21. Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. 2025. Evaltree: Profiling language model weaknesses via hierarchical capability trees. In Second Conference on Language Modeling. find
  22. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
  23. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  24. Qwen Team. 2025. Qwen3: Think Deeper, Act Faster \| Qwen. find
  25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108.
  26. Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Change Jia, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Bin Cui, Tong Yang, and Xiangzheng Zhang. 2025. Tinyr1-32b-preview: Boosting accuracy with branch-merge distillation. Preprin…
  27. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-skill-aware-data-selection-fine/