Elamur Research⌕
☰
home / papers / 2021 / Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

Paper · 2021 · Prakhar Ganesh, Yao Chen, Xin Lou et al.

Summary

Large pre-trained Transformers such as BERT deliver strong NLP results, but their size, memory footprint, and latency make them hard to run on phones, latency-sensitive services, or modest research hardware. This TACL 2021 survey reviews and compares compression techniques for Transformers, focusing on the original BERT. The authors first profile BERT BASE on a GPU and a CPU, measuring how size, FLOPs, memory, and latency are spread across its components. FFN sub-units are the main cost, while multi-head attention costs more on GPUs than theory predicts. They then classify methods such as quantization, pruning, knowledge distillation, replacing Transformer blocks with BiLSTMs or CNNs, and dynamic inference, and compare reported results. The main conclusion is that combining complementary methods, matched to the target hardware, works better than pushing one technique to its limit. The paper serves as a practical guide to building lighter BERT-style models and identifies open research directions.

Key points

  • Profiling setup: BERT BASE (L = 12, H = 768, A = 12) was profiled on 256-token inputs using the PyTorch profiler, on an Nvidia Titan X (12GB) and an Intel Xeon E5-1620 (32 GB). One example took 57.1 ms on the GPU and 750.9 ms on the CPU.
  • Quantization plus distillation: Zadeh et al. quantized a distilled BERT to 4 bits, shrinking it from 60.2% to 7.5% of the original size for an extra 0.9% accuracy loss. Sun et al. applied 8-bit quantization, cutting the size from 23% down to 5.25% while losing just 0.07% more accuracy.
  • Pruning plus distillation: Hou et al. combined distillation with structured pruning and reached 12.4% of the original size with a 0.96% accuracy drop. This beat structured pruning alone (Khetan and Karnin).
  • Hardware matters: Quantization and unstructured pruning reduce model size. They only speed up inference on specialized hardware or with specialized libraries.
  • Future directions: The authors suggest compressing each layer independently, exploring hybrid BiLSTM/CNN/Transformer designs, and further combining compression methods (e.g., Tambe et al.).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations

Also on the byline: Advanced Digital Sciences Center.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 77 works

Cite

Cite as Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen and Marianne Winslett. 2021. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00413

@article{ganesh2021compressing,
  title = {{Compressing Large-Scale Transformer-Based Models: A Case Study on BERT}},
  author = {Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen and Marianne Winslett},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00413},
  url = {https://doi.org/10.1162/tacl_a_00413}
}

References

82 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoonho Boo and Wonyong Sung. 2020. Fixed-point optimization of transformer neural network. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP ’20, pages 1753–1757. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr…
  3. Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. 2020. DeFormer: Decomposing pre-trained transformers for faster question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 4487–4497. find
  4. Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. 2020a. AdaBERT: Task-adaptive BERT compression with differentiable neural architecture search. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial I… — [doi:10.24963/ijcai.2020/341](https://doi.org/10.24963/ijcai.2020/341) find
  5. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michaelg Carbin. 2020b. The lottery ticket hypothesis for pre-trained BERT networks. In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS ’20, pages 1753–1757, Vancouver, Canad… find
  6. Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2018. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1):126–136. https://doi.org/10.1109/MSP.2017.2765695 — [doi:10.1109/msp.2017.2765695](https://doi.org/10.1109/msp.2017.2765695) ↗
  7. Yew Ken Chia, Sam Witteveen, and Martin Andrews. 2018. Transformer to CNN: Label-scarce distillation for efficient text classification. In Proceedings of the Compact Deep Neural Network Representation with Industrial Applications Workshop, Montréal, Canada. find
  8. Artem Chumachenko, Daniil Gavrilov, Nikita Balagansky, and Pavel Kalaidin. 2020. Weight squeezing: Reparameterization for extreme compression and fast inference. arXiv:2010. 06993. find
  9. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP’19, pages 276–286. Florence, Italy. https://doi .org/… find
  10. Kalyanmoy Deb. 2014. Multi-objective optimization. In Search Methodologies, pages 403–449. Springer. https://doi.org/10.1007/978 -1-4614-6940-7 15 — [doi:10.1007/978](https://doi.org/10.1007/978) ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Lifang Ding and Yujiu Yang. 2020. SDSK2BERT: Explore the specific depth with specific knowledge to compress BERT. In Proceedings of the IEEE International Conference on Knowledge Graph, ICKG ’20, pages 420–425. https://doi.org/10.1109/ICBK50248.2020.00066 ↗
  13. Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In Proceedings of the 8th International Conference on Learning Representations, ICLR ’20, Addis Ababa, Ethiopia.
  14. Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In Proceedings of the 10th International Conference on Learning Representations, ICLR ’21. find
  15. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681–694. https://doi.org/10.1007/s11023-020 -09548-1 — [doi:10.1007/s11023-020](https://doi.org/10.1007/s11023-020) ↗
  16. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the 7th International Conference on Learning Representations, ICLR ’19, New Orleans, LA, USA. find
  17. Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, RepL4NLP ’20, pages 143–155. https://doi.org/10 .18653/v1/2020.repl4nlp-1.18
  18. Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. PoWER-BERT: Accelerating BERT inference via progressive word-vector elimination. In Proceedings of the International Conference on Machine Learning, ICML ’20, pages 3690–3699. find
  19. Fu-Ming Guo, Sijia Liu, Finlay S. Mungall, Xue Lin, and Yanzhi Wang. 2019. Reweighted proximal pruning for large-scale language representation. arXiv:1909.12486. — [arXiv:1909.12486](https://arxiv.org/abs/1909.12486) find
  20. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 4365–…
  21. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. DynaBERT: Dynamic BERT with adaptive width and depth. In Advances in Neural Information Processing Systems, volume 33 of NeuIPS ’20. find
  22. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the 8th International Conference on Learning Representations, ICLR ’20, Addis Ababa, Ethiopia. find
  23. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 328–339, Melbourne, Australia. https://doi.org/10 .18653/v1/P18-1031 find
  24. Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. 2020a. Efficient transformer-based large scale language representations using hardware-friendly block structured pruning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3187–319… find
  25. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869–6898. find
  26. Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020b. BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 3009–3018. find
  27. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP ’20, pages 4163–4174. https:/… find
  28. Ashish Khetan and Zohar Karnin. 2020. schu-BERT: Optimizing elements of BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL’20, pages 2807–2818. https://doi/10.18653/v1/2020.acl-main.250 find
  29. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations, ICLR ’20, Addis Ababa, Ethiopia.
  30. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020c. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the International Conference on Machine Learning, ICML ’20, pages 5958–5968. find
  31. Zi Lin, Jeremiah Liu, Zi Yang, Nan Hua, and Dan Roth. 2020. Pruning redundant mappings in transformer models via spectral-normalized identity prior. In Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 719–730. https://doi.org/10.18653/v1/2020.findings -emnl… ↗
  32. Linqing Liu, Huan Wang, Jimmy Lin, Richard Socher, and Caiming Xiong. 2019a. MKD: A multi-task knowledge distillation approach for pretrained language models. arXiv:1911.03588. — [arXiv:1911.03588](https://arxiv.org/abs/1911.03588) find
  33. Brussels, Belgium. https://doi.org/10 .18653/v1/D18-1206 find
  34. Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: A self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 6035–6044.
  35. Matan Ben Noach and Yoav Goldberg. 2020. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL… find
  36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907. 11692. find
  37. Matous Machacek and Ondrej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT ’14, pages 293–301, Baltimore, MD, USA. https://doi.org/10.3115/v1/W14-3336 ↗
  38. Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. 2020. LadaBERT: Lightweight adaptation of BERT through hybrid model compression. In Proceedings of the 28th International Conference on Computational Linguistics, COLING ’20, pages 3225–3… find
  39. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32 of NeurIPS ’19, pages 14014–14024, Vancouver, BC, Canada.
  40. Subhabrata Mukherjee and Ahmed H. Awadallah. 2020. XtremeDistil: Multi-stage distillation for massive multilingual models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 2221–2234. https://doi.org/10.18653/v1/2020.acl-main.202 ↗
  41. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 1797–1807, Matthew Pe…
  42. Prafull Prakash, Saurabh Kumar Shashidhar, Wenlong Zhao, Subendhu Rongali, Haidar Khan, and Michael Kayser. 2020. Compressing transformer-based semantic parsing models using compositional code embeddings. In Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4… find
  43. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 3208–3229. https://doi.org/10 .18653/v1/2020.emnlp-main.259 find
  44. XiPeng Qiu, TianXiang Sun, YiGe Xu, YunFan Shao, Ning Dai, and XuanJing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897. https://doi.org/10.1007/s11431-020-1647-3 ↗
  45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67.
  47. Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33 of NeurIPS ’20. find
  48. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020. Fixed encoder self-attention patterns in transformer-based machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 556–568. https://doi.org/10.18653/v1/2020.findings-emnlp.49 ↗
  49. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-BERT: Hessian based ultra low precision quantization of BERT. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34 of AAAI ’20, pages 8815–8821. find
  50. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 784–789, Melbourne, Australia. https://doi.org/10.18653/v1/P18-2124 ↗
  51. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053. find
  52. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP ’16, pages 2383–2392, Austin, TX, USA. https://doi.org/10 .18653/v1/D…
  53. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl a 00349 ↗
  54. Corby Rosset. 2020. Turing-NLG: A 17-billion-parameter language model by Microsoft. Microsoft Research Blog, 2:13. find
  55. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. Poor man’s BERT: Smaller and faster transformer models. arXiv: 2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) find
  56. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, Vancouver, Canada. https://doi.org/10.1609/aaai.v34i05… ↗
  57. Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, and Tie-Yan Liu. 2020. LightPAFF: A two-stage distillation framework for pre-training and fine-tuning. arXiv:2004.12817. — [arXiv:2004.12817](https://arxiv.org/abs/2004.12817) find
  58. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages…
  59. Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020a. Contrastive distillation on intermediate representations for language model compression. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 498–508, https://doi .… find
  60. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020b. MobileBERT: A compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 2158–2170.
  61. Thierry Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En-Yu Yang, Marco Donato, Victor Sanh, Paul Whatmough, Alexander M Rush, David Brooks, and Gu-Yeon Wei. 2020. EdgeBERT: Sentence-level energy optimizations for latency-aware multi-task NLP inference. arXiv:2011.14203. find
  62. Raphael Tang, Yao Lu, and Jimmy Lin. 2019. Natural language generation for effective knowledge distillation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP, DeepLo ’19, pages 202–208, Hong Kong, China. https://doi.org/10.18653/v1/D19-6122 ↗
  63. Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2020. Synthesizer: Rethinking self-attention in transformer models. arXiv:2005.00743. find
  64. James Yi Tian, Alexander P. Kreuzer, Pai-Hung Chen, and Hans-Martin Will. 2019. WaL-DORf: Wasteless language-model distillation on reading-comprehension. arXiv:1912.06638. — [arXiv:1912.06638](https://arxiv.org/abs/1912.06638) find
  65. Henry Tsai, Jayden Ooi, Chun-Sung Ferng, Hyung Won Chung, and Jason Riesa. 2020. Finding fast transformers: One-shot neural architecture search by component composition. arXiv:2008.06808. — [arXiv:2008.06808](https://arxiv.org/abs/2008.06808) find
  66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS ’17, pages 6000–6010, Long Beach, CA, USA.
  67. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Blackbox-NLP ’…
  68. Shirui Wang, Wenan Zhou, and Chao Jiang. 2020a. A survey of word embeddings based on deep learning. Computing, 102(3):717–740. find
  69. Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b. Linformer: Self-attention with linear complexity. arXiv: 2006.04768. https://doi.org/10.1007/s00607-019-00768-7 ↗
  70. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020c. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33 of NeurIPS ’20. find
  71. Moshe Wasserblat, Oren Pereg, and Peter Izsak. 2020. Exploring the boundaries of low-resource BERT distillation. In Proceedings of the Workshop on Simple and Efficient Natural Language Processing, SustaiNLP ’20, pages 35–40. https://doi.org/10.18653/v1/2020 .sustainlp-1.5 ↗
  72. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  73. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 2246–2251. https://doi.org/10.18653/v1/2020.acl-main.204 ↗
  74. Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. BERT-of-Theseus: Compressing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 7859–7869.
  75. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32 of NeuIPS ’19, pages 5753–5763.
  76. Ali H. Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. 2020. GOBO: Quantizing attention-based NLP models for low latency and energy efficient inference. In Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’20, pages 811–824. https://doi .org/10.11… find
  77. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, Vancouver, Canada. https://doi.org/10.1109/EMC2 -NIPS53020.2019.00016 — [doi:10.1109/emc2](https://doi.org/10.1109/emc2) ↗
  78. Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. 2019a. Explicit sparse transformer: Concentrated attention through explicit selection. arXiv: 1912.11637. — [arXiv:1912.11637](https://arxiv.org/abs/1912.11637) find
  79. Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019b. Extreme language model compression with optimal subwords and shared projections. arXiv:1909.11687. — [arXiv:1909.11687](https://arxiv.org/abs/1909.11687) find
  80. Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoc Le, Qiang Liu, and Dale Schuurmans. 2020a. Go wide, then narrow: Efficient training of deep thin networks. In Proceedings of the International Conference on Machine Learning, ICML ’20, pages 11546–11555. find
  81. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020b. BERT loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33 of NeurIPS ’20. find
  82. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision, ICCV ’19,…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ganesh-2021-compressing-large-scale-transformer-based/