Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Paper · 2026 · Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba et al.
Summary
Standard knowledge distillation assumes that the teacher and student share one vocabulary, because the student learns by matching the teacher's next-token distribution. When the tokenizers differ, existing cross-tokenizer distillation (CTD) methods have to align the two vocabularies through heuristics, which adds complexity. The authors propose Byte-Level Distillation (BLD), a simple baseline that needs no alignment because bytes are a representation every tokenizer shares. The teacher's token probabilities are converted into byte-level probabilities using a fast approximation from Vieira et al. (2025). The student gets a small learnable byte-level head, placed alongside its usual token head, and distillation happens through this byte interface. Once training ends, the extra head is discarded.
The method was tested in three settings, with models from 1B to 8B parameters:
- moving Llama3.2 3B to the Qwen2 tokenizer;
- converting Llama3.2 3B to byte-level tokenization;
- distilling OpenMath2-Llama3.1-8B into Gemma2 2B.
BLD matches, and on some benchmarks beats, more elaborate methods such as MinED, ALM + SFT and DSKD. Its instruction following is clearly weaker, however. No method wins everywhere, so CTD remains an open problem.
Key points
- Architecture and training: The byte head consists of 10 parallel linear projections onto a 260-symbol vocabulary (256 bytes plus 4 special tokens). This lets each token position predict up to 10 bytes. The backbone is tuned with LoRA (rank 64, query/value projections only), and new embeddings are initialised with Fast Vocabulary Transfer (FVT).
- Llama3.2 3B → Qwen2 tokenizer: BLD scores best on PiQA (75.68) and AGI-ZH (35.97). Its IFEval score is only 30.58, versus 62.83 for MinED and 58.51 for ALM + SFT.
- BPE-to-byte transfer: This setting is much harder. All methods drop sharply, with MMLU about 21 points and ARC-C about 13 points below the original model. BLD leads only on PiQA (67.52), and only narrowly.
- OpenMath2-Llama3.1-8B → Gemma2 2B: BLD reaches 62.55 on GSM8K, ahead of ALM + SFT (61.56), SFT (59.29) and the base model (51.48). On MATH, SFT is better (22.40 vs. 20.08).
- Gap to the teacher: The teacher scores 87.26 on GSM8K and 37.60 on MATH, so every student still falls far short.
Institutions
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksTokenization & segmentation · 119 works
Cite
Cite as Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia and Davide Buffelli. 2026. Cross-Tokenizer LLM Distillation through a Byte-Level Interface. Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U).
@inproceedings{singh2026cross,
title = {{Cross-Tokenizer LLM Distillation through a Byte-Level Interface}},
author = {Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia and Davide Buffelli},
booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)},
year = {2026},
doi = {10.18653/v1/2026.customnlp4u-1.9},
url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.9}
}References
- Nicolas Boizard, Kevin El Haddad, CELINE HUDE-LOT, and Pierre Colombo. 2025. Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs. Transactions on Machine Learning Research.
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-cross-tokenizer-llm-distillation-through/