📄 Distilling Neural Networks for Greener and Faster Dependency Parsing

Summary

The paper looks for a way to make dependency parsing computationally cheaper and less harmful to the environment with almost no loss in quality. The authors took the Biaffine parser of Dozat and Manning, which combines high accuracy with speed, and compressed it using teacher-student distillation. A smaller student model is trained to approximate the probability distributions of the full teacher model over arcs and labels, using the Kullback–Leibler divergence. In parallel, the student also receives the standard loss function computed against the gold annotation. Experiments were conducted on nine typologically diverse UD treebanks, including Wolof, Uyghur and Ancient Greek, as well as on the English PTB. The models were compressed to 20, 40, 60 and 80% of the trainable parameters. The smallest variant loses on average only about one point of UAS and LAS. At the same time, it runs noticeably faster and consumes less energy. It is also considerably more accurate than a model of the same size trained without distillation. Distillation therefore yields real compression, not merely theoretical compression. This is useful for applying parsers at scale and for less powerful hardware.

Key points

• Accuracy–speed trade-off. The D-20 variant has about 2.4 million parameters, compared with roughly 12 million in the full model. On average it trails by about 1 point of UAS and LAS. Meanwhile, at inference time it runs 2.30 times faster on CPU and 1.19 times faster on GPU. For some treebanks the D-80 variant even slightly outperforms the baseline model.
• Data. The treebanks used are Ancient-Greek-Perseus, English-EWT, Russian-GSD, Tamil-TTB, Uyghur, Wolof, as well as Chinese, Finnish and Hebrew. For most languages FastText embeddings were used; for Ancient Greek and Wolof, embeddings from other sources. All embeddings were reduced to 100 dimensions.
• PTB. The D-40 model is more accurate than SeqLab, the fastest parser at the time (based on sequence labelling), by more than 1 point of UAS and LAS. D-20 is about four times faster than SeqLab on CPU but trails it by 0.62 UAS and 0.76 LAS.
• Energy. Parsing all test sets (about 8 thousand sentences) costs 0.32 kJ for the full model and 0.24 kJ for D-20.
• Training cost. On English-EWT, one distillation epoch takes 103.1 s, versus 66.4 s for the full model. However, with distillation the best result appears earlier: around epoch 50 rather than epochs 80–100. According to the authors' estimate, distillation pays off when the parser processes large volumes of text or runs for a long time.