Elamur Research⌕
☰
home / papers / 2026 / CTPD: Cross Tokenizer Preference Distillation

CTPD: Cross Tokenizer Preference Distillation

Paper · 2026 · Truong Nguyen, Phi Van Dat, Ngan Nguyen et al.

Summary

Small language models are cheaper to deploy but harder to align with human preferences. One remedy is to distill alignment from a larger, already-aligned teacher, so the expensive alignment step is paid only once. The richest form of this is white-box distillation, which uses the teacher's logits rather than its generated text. It breaks down, however, when teacher and student split text into different tokens. The only earlier white-box preference-distillation study assumed a shared vocabulary.

CTPD addresses the mismatched-tokenizer case. Its key observation is that both tokenizations describe the same characters. It therefore segments each sequence into spans where teacher and student token groups cover identical character intervals. Teacher log-probabilities are then mapped onto the student's spans without adding trainable parameters. On this basis, the authors move TIS-DPO's token importance weights to the span level and use the teacher itself as the DPO reference policy. They justify the design through importance sampling.

On six benchmarks, CTPD outperforms both preference-alignment and classic distillation baselines. This makes any strong teacher usable for aligning compact models, whatever its tokenizer.

Key points

  • Three components: Aligned Span Projection, a cross-tokenizer version of TIS-DPO with span-level importance weights, and a Teacher-Anchored Reference that uses the teacher as π_ref.
  • Evaluation: the Language Model Evaluation Harness on ARC, HellaSwag, Winogrande, MMLU, TruthfulQA and GSM8k.
  • Main results: average scores beat TIS-DPO by +1.26 and +0.66 points in the two reported settings. The largest gains are on GSM8k (+3.16) and TruthfulQA (+2.85).
  • Ablations (one table labelled Llama3.1-8B):
    • Random importance weights sharply hurt performance.
    • Length-averaged, student-estimated and teacher–student-estimated weights all fall short of the proposed scheme.
    • Using the student instead of the teacher as reference also lowers scores.
  • Training setup:
    • One epoch per stage with AdamW, batch size 16, on 8× NVIDIA H100-80GB.
    • CTPD uses lr 1e-6 and β = 0.1, with weights clipped to [−0.5, 1.5].
    • Code is available at github.com/dinhtruongng/CTPD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Efficient models & compression · 242 worksTokenization & segmentation · 119 works

Cite

Cite as Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le and Thanh Hong Nguyen. 2026. CTPD: Cross Tokenizer Preference Distillation. arXiv:2601.11865.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.11865 · arXiv 2601.11865

@misc{nguyen2026ctpd,
  title = {{CTPD: Cross Tokenizer Preference Distillation}},
  author = {Truong Nguyen, Phi {Van Dat}, Ngan Nguyen, Linh Ngo Van, Trung Le and Thanh Hong Nguyen},
  year = {2026},
  doi = {10.48550/arxiv.2601.11865},
  eprint = {2601.11865},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.11865}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.… find
  2. Beeching, E.; Lambert, N.; Tunstall, L.; Rajani, N.; and von Werra, L. 2023. Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/ open llm leaderboard. ↗
  3. Boizard, N.; El Haddad, K.; Hudelot, C.; and Colombo, P. 2024. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs.
  4. Christiano, P. F.; Leike, J.; Brown, T. B.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. find
  5. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457.
  6. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168.
  7. Cui, X.; Zhu, M.; Qin, Y.; Xie, L.; Zhou, W.; and Li, H. 2025. Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  8. Gao, S.; Wan, F.; Guo, J.; Quan, X.; and Wang, Q. 2025. Advantage-Guided Distillation for Preference Alignment in Small Language Models. arXiv:2502.17927. — [arXiv:2502.17927](https://arxiv.org/abs/2502.17927) find
  9. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300.
  10. Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531.
  11. Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-K.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; and Pfister, T. 2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. arXiv:2305.02301.
  12. Kloek, T.; and van Dijk, H. K. 1978. Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo. Econometrica: Journal of the Econometric Society, 46(1): 1–19. find
  13. Lin, S.; Hilton, J.; and Evans, O. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958.
  14. Liu, A.; Bai, H.; Lu, Z.; Sun, Y.; Kong, X.; Wang, S.; Shan, J.; Jose, A. M.; Liu, X.; Wen, L.; Yu, P. S.; and Cao, M. 2025. TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights. arXiv:2410.04350. — [arXiv:2410.04350](https://arxiv.org/abs/2410.04350) find
  15. Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101. find
  16. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  17. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instruction…
  18. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.
  19. Ren, M.; Zeng, W.; Yang, B.; and Urtasun, R. 2019. Learning to Reweight Examples for Robust Deep Learning. arXiv:1803.09050. — [arXiv:1803.09050](https://arxiv.org/abs/1803.09050) find
  20. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641. find
  21. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. find
  22. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. 2022. Learning to summarize from human feedback. arXiv:2009.01325. Sutawika, L.; Gao, L.; Schoelkopf, H.; Biderman, S.; Tow, J.; Abbasi, B.; ben fattori; Lovering, C.; farzanehnakhaee70;… find
  23. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830. find
  24. Zeng, Y.; Liu, G.; Ma, W.; Yang, N.; Zhang, H.; and Wang, J. 2024. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning. find
  25. Zhang, X.; Zhang, S.; Liang, Y.; Meng, F.; Chen, Y.; Xu, J.; and Zhou, J. 2025. A Dual-Space Framework for General Knowledge Distillation of Large Language Models. arXiv:2504.11426. — [arXiv:2504.11426](https://arxiv.org/abs/2504.11426) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-ctpd-cross-tokenizer-preference-distillation/