Sequence-Level Knowledge Distillation
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1139.pdf
Summary
High-quality neural machine translation (NMT) systems tend to be very large, which makes them slow and hard to deploy on ordinary hardware such as phones. Kim and Rush adapt knowledge distillation, where a compact "student" network learns from a big "teacher," to sequence-to-sequence translation using the attentional architecture of Luong et al. They first apply the standard word-level variant, which matches the teacher's per-token distributions. They then propose two sequence-level variants:
- Seq-KD retrains the student on the teacher's beam-search translations of the training set.
- Seq-Inter uses the beam candidate closest to the reference translation.
On English→German, a 2×500 LSTM student nearly matches a state-of-the-art 4×1000 teacher. A similar 2×500→2×100 compression works on a smaller dataset. The sequence-level students perform well with greedy decoding, so beam search becomes largely unnecessary. Adding magnitude-based weight pruning compresses the model further. This makes NMT practical on modest devices, and the authors suggest the method could carry over to other sequence-generation tasks.
Key points
- The best student decodes greedily about 10× faster than the teacher using beam search, with little quality loss. It beats a non-distilled baseline by 4.2 BLEU (greedy) and 1.7 BLEU (beam).
- The teacher's single top beam output carries a surprisingly large share of probability mass: on average 1.3% for German→English and 2.3% for Thai→English. This supports approximating the sequence distribution by its mode.
- Pruning 80% of the student's weights gives 13× fewer parameters than the teacher, at a cost of only 0.4 BLEU. Pruning 90% leaves about 8m parameters (26× fewer) and loses 1.0 BLEU, which suggests distillation and pruning are complementary.
- Word embeddings make up about 63% (50m of 84m) of the 2×500 English→German model's parameters. Temperature τ=1 worked best for distillation.
- Code is released as harvardnlp/seq2seq-attn, and a smartphone demo as harvardnlp/nmt-android.
Cited by
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Semi-Supervised Sequence Modeling with Cross-View Training
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- Non-Autoregressive Neural Machine Translation
- MiniLLM: On-Policy Distillation of Large Language Models
- Survey of Low-Resource Machine Translation
- DistiLLM: Towards Streamlined Distillation for Large Language Models
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
- Are Sixteen Heads Really Better than One?
- Massively Multilingual Transfer for NER
- The Curious Case of Hallucinations in Neural Machine Translation
- Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants
- Masked Language Model Scoring
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- Parsing Tweets into Universal Dependencies
- UniTE: Unified Translation Evaluation
- Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
- GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
Cite
@inproceedings{kim2016sequence,
title = {{Sequence-Level Knowledge Distillation}},
author = {Kim, Yoon and Rush, Alexander M.},
booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
year = {2016},
doi = {10.18653/v1/d16-1139},
eprint = {1606.07947},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d16-1139}
}References
- [Ba and Caruana2014] Lei Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep? In Proceedings of NIPS. [Bahdanau et al.2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of ICLR. [Bucil… find our card
- Fonollosa. 2016. Character-based Neural Machine Translation. arXiv:1603.00810. [Courbariaux et al.2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1. a… find our card
- [Liang et al.2006] Percy Liang, Alexandre Bouchard-Cote, Dan Klein, and Ben Taskar. 2006. An End-to-End Discriminative Approach to Machine Translation. In Proceedings of COLING-ACL. [Lin et al.2016] Zhouhan Lin, Matthieu Coubariaux, Roland Memisevic, and Yoshua Bengio. 2016. Neural Networks with Few… find
- [Rush et al.2015] Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of EMNLP. [See et al.2016] Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of Neural Machine Translation via Pruning… find