Elamur Research⌕
☰
home / papers / 2026 / GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning

GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning

Paper · 2026 · Kasidit Sermsri and Teerapong Panboonyuen

Summary

GateKD addresses a weakness in distilling multi-step reasoning from large language models into small student models. Existing methods, including the mentor-based Mentor-KD, pass teacher signals to the student without checking whether the teacher is reliable on a given input. As a result, hallucinated rationales and unstable internal patterns are copied into students that are too small to correct them. The authors make distillation adaptive. They estimate teacher confidence from predictive entropy and use it to decide how much the student learns from three sources: the teacher's soft labels, its hidden states, and its attention maps. Uncertain signals are down-weighted, so training becomes a feedback loop rather than one-way copying. The method was tested on commonsense, logical, and symbolic reasoning benchmarks with T5 and Flan-T5 students of several sizes. According to the authors, it beats strong open-loop baselines, with the largest gains on logical and symbolic tasks. It also stays robust when distillation data is scarce. This matters for building compact reasoners that are cheap to run and less prone to inheriting errors.

Key points

  • The framework has three gated pathways: confidence-gated soft supervision, gated hidden-state evolution, and reliability-filtered attention distillation. All three share a single gate based on predictive entropy.
  • The work builds on Mentor-KD (Lee et al., 2024) but shifts the question. Mentor-KD asks where supervision should come from, while GateKD asks when and how it should be transferred.
  • Students use T5 and Flan-T5 backbones. In ablations, removing any one gating component lowers performance, which suggests the three parts work together. The extracted text did not include the exact scores.
  • In a qualitative StrategyQA example, the teacher writes a fluent but speculative rationale and answers wrongly. GateKD detects the teacher's low confidence and favors the mentor's constraint-aware reasoning instead.
  • The authors list several limitations:
    • Entropy-based confidence may be miscalibrated, especially out of distribution.
    • The method needs access to teacher internals, so it cannot be applied directly to API-only models.
    • Training costs more, although inference does not.
    • Evaluation covers English benchmarks only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chulalongkorn UniversityBangkok · Thailand6 works · 304 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEfficient models & compression · 242 works

Cite

Cite as Kasidit Sermsri and Teerapong Panboonyuen. 2026. GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.49

@inproceedings{sermsri2026gatekd,
  title = {{GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning}},
  author = {Kasidit Sermsri and Teerapong Panboonyuen},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.49},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.49}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. General Chen et al. 2025a. Effectiveness of chain-of-thought in distilling reasoning capability from large teacher models. In Proceedings of the 2025 International Natural Language Generation Conference, pages 49–60. find
  2. Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, et al. 2025b. Unveiling the key factors for distilling chain-of-thought reasoning. pages 15094– 15119. find
  3. Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024. Improve student’s reasoning generalizability through cascading decomposed cots distillation. arXiv preprint, arXiv:2405.19842. find
  4. Yuntian Deng, Kiran Prasad, Roland Fernandez, et al. 2023. Implicit chain of thought reasoning via knowledge distillation. In ArXiv preprint. find
  5. Nathan Ho, Madhu Shridhar, et al. 2023. Teaching small language models to reason. In ArXiv preprint.
  6. Alex Hsieh et al. 2023. Rationale-based knowledge distillation for neural reasoning. Artificial Intelligence Review. Review of rationale distillation techniques for LLMs. find
  7. Hojae Lee, Junho Kim, and SangKeun Lee. 2024. Mentor-kd: Making small language models better multi-step reasoners. arXiv preprint arXiv:2410.09037. find
  8. Kyumin Lee, Minjin Jeon, Sanghwan Jang, and Hwanjo Yu. 2025. Steper: Step-wise knowledge distillation for enhancing reasoning ability in multi-step retrieval-augmented language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29501–29523. Assoc… find
  9. Zhenyu Lei, Zhen Tan, Song Wang, et al. 2025. Learning from diverse reasoning paths with routing and collaboration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2832–2845. Association for Computational Linguistics.
  10. Patomporn Payoungkhamdee, Peerat Limkonchotiwat, Jinheon Baek, et al. 2024. An empirical study of multilingual reasoning distillation for question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7739–7751. Association for Computational Ling… find
  11. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. 2025. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint, arXiv:2502.21074.
  12. Author Song et al. 2025. Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions. Artificial Intelligence Review. find
  13. Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint, arXiv:2203.11171. Published at ICLR 2023.
  14. Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems.
  15. Shicheng Xu, Liang Pang, Yunchang Zhu, et al. 2025. Distilling the implicit multi-branch structure in llms’ reasoning via reinforcement learning. arXiv preprint, arXiv:2505.16142. — [arXiv:2505.16142](https://arxiv.org/abs/2505.16142) find
  16. Denny Zhou, Nathanael Schärli, Le Hou, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. In arXiv preprint.
  17. Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, and Bowen Zhou. 2023. Pad: Program-aided distillation can teach small models reasoning better than chain-of-thought fine-tuning. arXiv preprint, arXiv:2305.13888. find
  18. Chenglin Li, Qianglong Chen, Liangyue Li, Caiyu Wang, Yicheng Li, et al. 2024a. Mixed distillation helps smaller language models reason better. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 375–392. Association for Computational Linguistics. find
  19. Xiang Li, Shizhu He, Jiayu Wu, Zhao Yang, Yao Xu, et al. 2024b. Mode-cotd: Chain-of-thought distillation for complex reasoning tasks with mixture of decoupled lora-experts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics (LREC-COLING), pages 11475–11485. ELRA. find
  20. OpenAI. 2023. Gpt-4 technical report. arXiv preprint, arXiv:2303.08774.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sermsri-2026-gatekd-confidence-gated-closed-loop/