TinyBERT: Distilling BERT for Natural Language Understanding (2020 · 2,378 citations)
Large pre-trained language models such as BERT achieve very high performance. However, they have many parameters and slow inference, which makes them difficult to run on phones and other low-resource devices. The authors compress BERT with a knowledge distillation method designed specifically for the Transformer…
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020 · 213 citations)
Fine-tuned BERT models are large: BERT-base has 109M parameters, over 400 MB in 32-bit floats. This makes them hard to run on phones and other constrained hardware. Earlier quantization work kept accuracy at 8 bits. Going lower usually needed mixed precision, which some hardware handles poorly, or product…

Co-authors here

8 co-authors with a page, the most papers together first.

Venues

Where their works here were published, the most works first, with the editions of those that have a page.

Papers of our library cite their works 714 times (of the 149,439 papers whose bibliographies we read); 75 citing papers with a page here →.

Updated 30 October 2026