Elamur Research⌕
☰
home / papers / 2024 / DDK: Distilling Domain Knowledge for Efficient Large Language Models

DDK: Distilling Domain Knowledge for Efficient Large Language Models

Paper · 2024 · Jiaheng Liu, Chenchen Zhang, Jinyang Guo et al.

Summary

Smaller LLMs are cheaper to deploy, and knowledge distillation from a stronger model can improve them. However, existing LLM distillation methods train on a fixed data mix. They overlook that the gap between student and teacher differs widely by domain. For example, a Qwen-1.5 1.8B student trails its 14B teacher far more on Books than on The Stack. DDK addresses this by reshaping the distillation corpus during training. It splits the corpus into domains such as CommonCrawl, C4, Wikipedia, ArXiv, StackExchange, OpenWebMath and Chinese sources. It then periodically compares student and teacher perplexity on held-out validation sets for each domain. A softmax over these perplexity ratios sets the sampling probabilities, so domains where the student lags most receive more data. A smoothing rule for updating these weights keeps the data mixture from swinging abruptly, and the student is trained to match the teacher's logits. Across Qwen-1.5 and StarCoder teacher–student pairs, DDK consistently beats continued pretraining, standard KD and MiniLLM. This makes domain-aware data mixing a practical tool for building compact models.

Key points

  • Domain discrepancy factor: each domain's factor is a softmax over the student-to-teacher perplexity ratio. It is recomputed at intervals and drives domain-wise sampling, and a factor smooth updating step stabilizes it.
  • Qwen-1.5 14B → 4B: DDK reaches an average of 63.58 across 11 benchmarks, compared with 61.50 for KD, 61.13 for MiniLLM and 60.38 for CPT. Its largest gains are on GSM8K (62.09 vs. 56.00 for KD) and MBPP.
  • Qwen-1.5 7B → 1.8B: DDK averages 53.64, ahead of MiniLLM at 52.07. In 5-shot evaluation with a 14B teacher and 1.8B student, DDK scores 60.36 versus 57.48 for TED.
  • StarCoder 15.5B → 3B on CrossCodeEval: DDK raises average EM/ES to 37.6/69.4, compared with 34.6/61.2 for KD and 28.1/52.4 for the original student.
  • Convergence and ablation: most of the improvement arrives within roughly the first 10–15B training tokens, after which performance plateaus. Removing factor smoothing makes the per-domain discrepancy curves unstable and lowers results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu and Bo Zheng. 2024. DDK: Distilling Domain Knowledge for Efficient Large Language Models. arXiv:2407.16154.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.16154 · arXiv 2407.16154

@misc{liu2024ddk,
  title = {{DDK: Distilling Domain Knowledge for Efficient Large Language Models}},
  author = {Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu and Bo Zheng},
  year = {2024},
  doi = {10.48550/arxiv.2407.16154},
  eprint = {2407.16154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.16154}
}

References

66 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023. find
  2. I. Amos, J. Berant, and A. Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PdaPky8MUn. ↗
  3. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. URL https://arxiv.org/abs/2305.10403. ↗
  4. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.
  5. G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, and W. Ouyang. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv, 2024.
  6. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B.… — Qwen Technical Report
  7. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. URL https://arxiv.org/abs/ 2204.05862. ↗
  8. Baichuan. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. URL https://arxiv.org/abs/2309.10305. ↗
  9. T. Brown, B. Mann, N. Ryder, M. Subbiah, et al. Language models are few-shot learners. In Proceedings of NeurIPS, 2020. URL https://papers.nips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. ↗
  10. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… — Evaluating Large Language Models Trained on Code
  11. Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson. DISCO: Distilling counterfactuals with large language models. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… find
  12. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL https://arxiv.org/abs/2210.11416. ↗
  13. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  15. T. Computer. Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  16. T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
  17. Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in… find
  18. X. Du, Z. Yu, S. Gao, D. Pan, Y. Cheng, Z. Ma, R. Yuan, X. Qu, J. Liu, T. Zheng, X. Luo, G. Zhou, B. Yuan, W. Chen, J. Fu, and G. Zhang. Chinese tiny llm: Pretraining a chinese-centric large language model, 2024.
  19. S. Fan, M. Pagliardini, and M. Jaggi. DOGE: Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endedness Workshop, 2023. URL https://openreview.net/forum?id=qiKqsqwYXm. Bard, 2023. URL ↗
  20. Google. bard-google-ai-search-updates/. https://blog.google/technology/ai/ ↗
  21. J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819, 2021. find
  22. Y. Gu, L. Dong, F. Wei, and M. Huang. Knowledge distillation of large language models, 2024. find
  23. H. Guo, J. Yang, J. Liu, L. Yang, L. Chai, J. Bai, J. Peng, X. Hu, C. Chen, D. Zhang, et al. Owl: A large language model for it operations. arXiv preprint arXiv:2309.09298, 2023. — [arXiv:2309.09298](https://arxiv.org/abs/2309.09298) find
  24. J. Guo, J. Wu, Z. Wang, J. Liu, G. Yang, Y. Ding, R. Gong, H. Qin, and X. Liu. Compressing large language models by joint sparsification and quantization. ICML, 2024. find
  25. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  26. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  27. N. Ho, L. Schmid, and S.-Y. Yun. Large language models are reasoning teachers. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, Toronto, Canada, July 2023. Associ…
  28. C.-Y. Hsieh, C.-L. Li, C.-k. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association…
  29. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.
  30. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
  31. Y. Jiang, C. Chan, M. Chen, and W. Wang. Lion: Adversarial distillation of proprietary large language models, 2023.
  32. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. ↗
  33. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  34. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017.
  35. Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.
  36. C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao. Less is more: Task-aware layer-wise distillation for language model compression. ICML, 2023. find
  37. J. Liu, Z. Bai, Y. Zhang, C. Zhang, Y. Zhang, G. Zhang, J. Wang, H. Que, Y. Chen, W. Su, et al. E2-llm: Efficient and extreme length extension of large language models. arXiv preprint arXiv:2401.06951, 2024. find
  38. OpenAI. OpenAI: Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. ↗
  39. OpenAI. GPT-4 technical report, 2023. URL https://cdn.openai.com/papers/gpt-4. pdf. ↗
  40. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of NeurIPS, 2022. URL https://arxiv.org/abs/2203.02155. ↗
  41. S. Padmanabhan, Y. Onoe, M. J. Zhang, G. Durrett, and E. Choi. Propagating knowledge updates to lms through distillation. arXiv preprint arXiv:2306.09306, 2023. find
  42. K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
  43. B. Peng, C. Li, P. He, M. Galley, and J. Gao. abs/2304.03277, 2023. Instruction tuning with gpt-4. ArXiv,
  44. H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu, Y. Ma, F. Duan, Z. Bai, J. Wang, Y. Zhang, X. Tan, J. Fu, W. Su, J. Wang, L. Qu, and B. Zheng. D-cpt law: Domain-specific continual pre-training scaling law for large language models. ArXiv, abs/2406.01375, 2024. find
  45. A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. URL https://api.semanticscholar.org/CorpusID:49313245. ↗
  46. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  47. K. Sun, D. Yu, D. Yu, and C. Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 2020. URL https://arxiv.org/abs/1904.09679v3. ↗
  48. T. Sun, L. Chai, Y. Y. Jian Yang, H. Guo, J. Liu, B. Wang, L. Yang, and Z. Li. Unicoder: Scaling code large language model via universal code. ACL, 2024. find
  49. R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999. find
  50. A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018.
  51. I. Timiryasov and J.-L. Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. arXiv preprint arXiv:2308.02019, 2023. find
  52. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv.org/pdf/2302.13971.pd… ↗
  53. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023.
  54. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  55. P. Wang, Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren. Scott: Self-consistent chain-of-thought distillation. In Annual Meeting of the Association for Computational Linguistics, 2023.
  56. Z. M. Wang, Z. Peng, H. Que, J. Liu, W. Zhou, Y. Wu, H. Guo, R. Gan, Z. Ni, M. Zhang, Z. Zhang, W. Ouyang, K. Xu, W. Chen, J. Fu, and J. Peng. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746, 2023.
  57. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In Proceedings of ICLR, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR. ↗
  58. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. URL https://openreview.net/pdf?id=yzkSU5zdwD. ↗
  59. Y. Wu, J. Liu, X. Bu, J. Liu, Z. Zhou, Y. Zhang, C. Zhang, Z. Bai, H. Chen, T. Ge, W. Ouyang, W. Su, and B. Zheng. Conceptmath: A bilingual concept-wise benchmark for measuring mathematical reasoning of large language models. arXiv, 2024. find
  60. S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024. find
  61. Z. Yang, A. Zeng, Z. Li, T. Zhang, C. Yuan, and Y. Li. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17185–17194, 2023. find
  62. J. Ye, P. Liu, T. Sun, Y. Zhou, J. Zhan, and X. Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. find
  63. C. Zhang, D. Song, Z. Ye, and Y. Gao. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.07052, 2023.
  64. G. Zhang, S. Qu, J. Liu, C. Zhang, C. Lin, C. L. Yu, D. Pan, E. Cheng, J. Liu, Q. Lin, R. Yuan, T. Zheng, W. Pang, X. Du, Y. Liang, Y. Ma, Y. Li, Z. Ma, B. Lin, E. Benetos, H. Yang, J. Zhou, K. Ma, M. Liu, M. Niu, N. Wang, Q. Que, R. Liu, S. Liu, S. Guo, S. Gao, W. Zhou, X. Zhang, Y. Zhou, Y. Wang,… find
  65. P. Zhang, G. Zeng, T. Wang, and W. Lu. Tinyllama: An open-source small language model, 2024.
  66. Z. Zhou, J. Liu, C. Yang, J. Shao, Y. Liu, X. Yue, W. Ouyang, and Y. Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023. — [arXiv:2310.03708](https://arxiv.org/abs/2310.03708) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2024-ddk-distilling-domain-knowledge-efficient/