DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Paper · 2025 · Jongwoo Ko, Tianyi Chen, Sungnyun Kim et al.
Summary
Knowledge distillation (KD) for LLMs usually applies the same divergence to teacher-generated outputs (TGOs) and student-generated outputs (SGOs). The authors argue this ignores how the choice of loss interacts with the type of data. A naive contrastive alternative, DPKD, plugs the teacher into DPO in place of the reference model. They find it prone to reward hacking: it pushes student-response likelihood down excessively and produces degenerate text.
DistiLLM-2 extends DistiLLM with three components:
- Contrastive objective: skew KL (SKL) on teacher responses raises their likelihood, while skew reverse KL (SRKL) on the student's own responses lowers theirs.
- Data curation: SGOs are collected once per epoch in batches, which allows fast generation with vLLM.
- Curriculum: the skew coefficient α adapts per sample, and the SRKL weight β grows over training.
The method reaches state-of-the-art results for small models on instruction-following, math reasoning and code generation. It also transfers to other uses: building reference models for preference alignment, vision-language distillation, recovering quantized models, and training speculative-decoding drafters. This makes it a practical recipe for building compact models.
Key points
- The training loss is a β-weighted mix of SKL (computed on teacher responses) and SRKL (computed on student responses). The paper was published at ICML 2025, and the code is at github.com/jongwooko/distillm-2.
- The DPKD failure is illustrated by negative log-likelihood. Student responses reached an NLL of 91.25, while teacher responses were fitted only to 20.29.
- For VLM distillation, the teacher was LLaVA-1.5-7B and the student TinyLLaVA-1.4B, trained on 83K prompts from RLAIF-V-Dataset. On OK-VQA and TextVQA, DistiLLM-2 averaged +2.01% over GKD and +4.61% over DistiLLM.
- In speculative decoding, a Llama-68m drafter trained with DistiLLM-2 delivered the highest speedups. It reached about ×1.97 with Phi3-medium and ×1.84 with Phi3.5-mini as verifiers, beating drafters trained with SFT, GKD and DistiLLM.
- For a 4-bit quantized Phi-3.5-mini, LoRA-based recovery via DistiLLM-2 gave the best average score among the KD baselines.
Fields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works
🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works
Cite
Cite as Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov and Se-Young Yun. 2025. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. arXiv:2503.07067.
@misc{ko2025distillm,
title = {{DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs}},
author = {Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov and Se-Young Yun},
year = {2025},
doi = {10.48550/arxiv.2503.07067},
eprint = {2503.07067},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2503.07067}
}References
- This paper presents work whose goal is to advance the field of machine learning, specifically efficiency and efficacy of LLMs-based system. We also have shown four possible applications (but not limited to) as follows: Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree,… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/ko-2025-distillm-2-contrastive-approach-boosts/