Elamur Research⌕
☰
home / papers / 2024 / Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

Summary

Logit-based knowledge distillation usually relies on KL divergence. This requires the teacher and the student to share one vocabulary, but LLM families rarely share tokenizers. As a result, practitioners mostly fall back on fine-tuning the student on text the teacher has generated. Drawing on optimal transport, the authors introduce a new objective they name the Universal Logit Distillation (ULD) loss. It uses a closed-form Wasserstein distance to compare the output distributions of teacher and student even when their vocabularies differ. The method treats both models as black boxes, needing only their logits, so it also works across architectures. They test it with a LLama teacher and smaller students such as Pythia-410m, Bloomz-560m and the encoder-decoder MT0-580m. Tasks cover extractive QA, generative QA and summarization, on SQuAD, QED, PubMedQA and DIALOGSum. ULD consistently beats distillation from teacher-generated text. It reaches comparable quality with roughly half the data or half the student size, and it curbs overfitting. Code, weights and generated datasets are released, making distillation feasible between arbitrary model families.

Key points

  • Tokenizer mismatch is substantial: for example, the Bloomz tokenizer covers only 30.03% of Mistral's vocabulary, so KL-based logit distillation cannot be applied directly.
  • In the dataset-size ablation (LLama → Pythia-410m, five runs per ratio), ULD wins at every ratio tested (25%, 50%, 75% and 100% of the data). With only 50% of the data, it outperforms teacher-text training on the full set.
  • Pythia-410m trained with ULD matches Pythia-1b trained on teacher-generated text on QED and DIALOGSum.
  • Cross-architecture transfer (LLama decoder → MT0-580m encoder-decoder), teacher-text vs ULD:
    • QED F1: 55.63 → 56.01
    • PubMedQA BERTScore: 27.56 → 30.19
    • DIALOGSum Rouge-Lsum: 23.22 → 23.92
  • Resources: code is at github.com/Nicolas-BZRD/llm-recipes, and models and datasets are on Hugging Face (Nicolas-BZRD).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics Language models & pretraining Tokenization & segmentation Efficient models & compression

 Institutions

🎓Université Paris-SaclayGif-sur-Yvette · France26 works · 5,075 citations🔬CentraleSupélecGif-sur-Yvette · France7 works · 1,242 citations🔬Mathématiques et Informatique pour la Complexité et les SystèmesGif-sur-Yvette2 works · 260 citations

Also on the byline: University of Mons.

Cite

Cite as Nicolas Boizard, Kevin El Haddad, Celine Hudelot and Pierre Colombo. 2024. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs. arXiv:2402.12030 · Transactions on Machine Learning Research, January 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.12030 · arXiv 2402.12030

@misc{boizard2024cross,
  title = {{Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs}},
  author = {Nicolas Boizard, Kevin {El Haddad}, Celine Hudelot and Pierre Colombo},
  year = {2024},
  doi = {10.48550/arxiv.2402.12030},
  eprint = {2402.12030},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12030}
}

References

66 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models.… our card
  2. Hareesh Bahuleyan, Lili Mou, Hao Zhou, and Olga Vechtomova. Stochastic wasserstein autoencoder for probabilistic sentence generation. 2018. find
  3. Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. Knot: Knowledge distillation using optimal transport for solving nlp tasks, 2022. find
  4. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… find
  5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive… our card
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. URL https://arxiv.org/abs/1607.04606. ↗ our card
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. volume 33, pp. 1877–1901, 2020. our card
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. our card
  9. Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pp. 535–541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: 1… — [doi:10.1145/1150402.1150464](https://doi.org/10.1145/1150402.1150464) find
  10. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021. our card
  11. Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. Automatic text evaluation through the lens of Wasserstein barycenters. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Langu… find
  12. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. volume 26, 2013. find
  13. Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. Cost-effective distillation of large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 7346–7354, 2023. find
  14. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. find
  15. Zilin Du, Haoxin Li, Xu Guo, and Boyang Li. Training on synthetic data beats real data in multimodal relation extraction, 2023. find
  16. Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Segu… find
  17. Nuno M Guerreiro, Pierre Colombo, Pablo Piantanida, and André FT Martins. Optimal transport for unsupervised hallucination detection in neural machine translation. 2022. find
  18. Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, Zhaohui Hou, Zhiyuan Huang, Shaoqing Lu, Ding Liang, and Mingjie Zhan. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise. Number arXiv:2310.19019.… — [doi:10.48550/arxiv.2310.19019](https://doi.org/10.48550/arxiv.2310.19019) find
  19. Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. Generate, Annotate, and Learn: NLP with Synthetic Text. volume 10, pp. 826–842, 08 2022. doi: 10.1162/tacl_a_00492. URL https://doi.org/10.1162/tacl_a_00492. ↗
  20. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Number arXiv:1503.02531arXiv:1503.02531. arXiv, March 2015. doi: 10.48550/arXiv.1503.02531. URL http://arxiv.org/abs/1503.02531. arXiv:1503.02531 [cs, stat]. ↗ our card
  21. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes, 2023. our card
  22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  23. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  24. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020. find
  25. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. our card
  26. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, B… find
  27. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. do… our card
  28. Alina Kramchaninova and Arne Defauw. Synthetic data generation for multilingual domain-adaptable question answering systems. In Helena Moniz, Lieve Macken, Andrew Rufener, Loïc Barrault, Marta R. Costa-jussà, Christophe Declercq, Maarit Koponen, Ellie Kemp, Spyridon Pilos, Mikel L. Forcada, Carolina… find
  29. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. Qed: A framework and dataset for explanations in question answering, 2020. find
  30. VI Lcvenshtcin. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet Physics-Doklady, volume 10, 1966. find
  31. Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant. Reducing retraining by recycling parameter-efficient prompts, 2022. find
  32. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023. find
  33. Tianda Li, Yassir El Mesbahi, Ivan Kobyzev, Ahmad Rashid, Atif Mahmud, Nithin Anchuri, Habib Hajimo-lahoseini, Yang Liu, and Mehdi Rezagholizadeh. dis, 2021. find
  34. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. ↗ our card
  35. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. our card
  36. Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, and Laurent Besacier. Small-100: Introducing shallow multilingual machine translation model for low-resource languages. Number arXiv:2210.11621arXiv:2210.11621. arXiv, October 2022. doi: 10.48550/arXiv.2210… find
  37. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel… our card
  38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… our card
  39. Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. volume 11, pp. 355–607. Now Publishers, Inc., 2019. find
  40. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. our card
  41. Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=2fc5GOPYip. ↗
  42. Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. 2023. our card
  43. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. find
  44. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… our card
  45. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation, 2021. our card
  46. Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz. Beyond accuracy, f-score and roc: A family of discriminant measures for performance evaluation. volume Vol. 4304, pp. 1015–1021, 01 2006. ISBN 978-3-540-49787-5. doi: 10.1007/11941439_114. — [doi:10.1007/11941439_114](https://doi.org/10.1007/11941439_114) find
  47. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020. our card
  48. Inar Timiryasov and Jean-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty, 2023. find
  49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. Number arXiv:2… our card
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  51. Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. An empirical comparison of lm-based question and answer generation methods. 2023. find
  52. Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009. find
  53. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020a. find
  54. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021. find
  55. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. volume 53, pp. 1–34. ACM New York, NY, USA, 2020b. find
  56. Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-lm: A diverse herd of distilled models from large-scale instructions, 2023. find
  57. Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for word embedding and topic modeling, 2018. find
  58. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. Fantastic questions and where to find them: Fairytaleqa – an au… our card
  59. Jianbo Ye, Panruo Wu, James Z. Wang, and Jia Li. Fast discrete distribution clustering using wasserstein barycenter with sparse support, 2017. find
  60. Qinyuan Ye, Iz Beltagy, Matthew Peters, Xiang Ren, and Hannaneh Hajishirzi. FiD-ICL: A fusion-in-decoder approach for efficient in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics… find
  61. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… our card
  62. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr. ↗
  63. Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay-Yoon Lee, Mohit Iyyer, and Andrew McCallum. Multistage collaborative knowledge distillation from large language models. Number arXiv:2311.08640. arXiv, November 2023a. URL http://arxiv.org/abs/2311.08640. arXiv:231… — [arXiv:2311.08640](https://arxiv.org/abs/2311.08640) ↗
  64. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded dat… find
  65. Tianxun Zhou and Keng-Hwee Chiam. Synthetic data generation method for data-free knowledge distillation in regression neural networks. volume 227, pp. 120327. Elsevier, 2023. find
  66. Yubo Zhuang, Xiaohui Chen, and Yun Yang. Wasserstein k-means for clustering probability distributions, 2022. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boizard-2024-towards-cross-tokenizer-distillation-universal/