Elamur Research⌕
☰
home / papers / 2020 / HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

Paper · 2020 · Hanrui Wang, Zhanghao Wu, Zhijian Liu et al.

Summary

Transformer models for machine translation are too slow on constrained devices. For example, Transformer-Big needs about 13G FLOPs and roughly 20 seconds on a Raspberry Pi to translate a 30-word sentence. The authors make two observations: FLOPs poorly predict real latency, and the architecture that runs fastest differs from one device to another. Their framework, HAT, therefore puts measured hardware latency directly into a neural architecture search.

The search space is broad. Each decoder layer may attend to several encoder layers, not only the last one, and every layer can have its own embedding size, hidden size and number of heads. A single weight-shared SuperTransformer is trained by sampling random SubTransformers, and its inherited weights serve as a cheap estimate of each candidate's quality. A per-device latency predictor, fitted on measured (architecture, latency) pairs, guides an evolutionary search under a latency budget. The winning model is then retrained from scratch.

Across four translation benchmarks and three devices, HAT yields faster and smaller models without losing BLEU. Its search costs a tiny fraction of the Evolved Transformer's, which makes device-specific model design practical.

Key points

  • Evaluation covers WMT'14 En-De, WMT'14 En-Fr, WMT'19 En-De and IWSLT'14 De-En, on a Raspberry Pi ARM CPU, an Intel Xeon CPU and an Nvidia TITAN Xp GPU.
  • On WMT'14 with a Raspberry Pi-4, HAT is 3× faster and 3.7× smaller than Transformer-Big.
    • Against the Evolved Transformer, it is 2.7× faster and 3.6× smaller.
    • Its search is 12,041× cheaper than the Evolved Transformer's, which reportedly needed about 250 GPU years.
  • At matching BLEU, HAT runs as much as 1.9× faster than both the Levenshtein Transformer and the Lite Transformer.
  • Specialization matters, as shown on WMT'14 En-De:
    • The GPU-tuned model runs at 147 ms on GPU but 6491 ms on ARM.
    • The ARM-tuned model runs at 6042 ms on ARM but 184 ms on GPU.
  • HAT combines well with compression:
    • K-means 4-bit quantization without finetuning shrinks the WMT'14 En-Fr model 25× (705MB → 28MB) at 41.1 vs 41.2 BLEU.
    • Knowledge distillation lifts a 30M-parameter student from 25.8 to 26.1 BLEU.

Interesting ideas

  • I1. Measure latency instead of counting FLOPs. A small predictor trained on real timings from the target device gives fast, accurate feedback during model selection. This suits any project that ships models to phones, edge boxes or servers with different profiles.
  • I2. Train one supernet, then search many times. A weight-shared SuperTransformer is trained once and then reused to rank architectures for any device or latency budget. The expensive step happens only once, so new deployment targets become cheap to add.
  • I3. Let the decoder see all encoder levels. Letting decoder layers attend to concatenated outputs of several encoder layers removes the bottleneck of using only the top layer. This gives the decoder access to lower-level features and could help other encoder-decoder setups, such as tagging or seq2seq for low-resource languages.
  • I4. Allow layers to differ in shape. Giving each layer its own width and head count exposes hardware-specific trade-offs. In the search results, GPUs favored shallow, wide models, while ARM CPUs favored deep, thin ones. This is a useful rule of thumb when choosing model shapes by hand.
  • I5. Combine architecture search with compression. Since architecture search and post-hoc compression (quantization, distillation) are independent, they can be stacked. Cheap K-means quantization without finetuning made a 25× size reduction possible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🏢IBMArmonk · United States85 works · 66,118 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 GPU & parallel computing · 7 works

Cite

Cite as Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan and Song Han. 2020. HAT: Hardware-Aware Transformers for Efficient Natural Language Processing. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.686

@inproceedings{wang2020hat,
  title = {{HAT: Hardware-Aware Transformers for Efficient Natural Language Processing}},
  author = {Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan and Song Han},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.686},
  url = {https://doi.org/10.18653/v1/2020.acl-main.686}
}

References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. 2019. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1269–1281, Florence, Italy. Association for Computational Linguistics. find
  2. Ankur Bapna, Mia Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028–3033, Brussels, Belgium. Association for Computation… find
  3. Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 550–559, Stockholm… find
  4. Han Cai, Chuang Gan, and Song Han. 2019a. Once for all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791. find
  5. Han Cai, Ligeng Zhu, and Song Han. 2019b. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations. find
  6. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in ne… our card
  7. Jason Cong, Zhenman Fang, Michael Lo, Hanrui Wang, Jingxian Xu, and Shaochong Zhang. 2018. Understanding performance differences of fpgas and gpus. In 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 93–96. IEEE. find
  8. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243–1252, International Convention… our card
  9. Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for GPUs. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1302–1310, International Conventio… our card
  10. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations. our card
  11. Jiatao Gu, Changhan Wang, and Jake Zhao. 2019. Levenshtein Transformer. arXiv. find
  12. Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2019. Single path one-shot neural architecture search with uniform sampling. find
  13. Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al. 2020. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architec… find
  14. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784–800. find
  15. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. our card
  16. Marcin Junczys-Dowmunt. 2018. Microsoft’s submission to the wmt2018 news translation task: How i learned to stop worrying and love the data. arXiv preprint arXiv:1809.00196. find
  17. Marcin Junczys-Dowmunt. 2019. Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation. arXiv preprint arXiv:1907.06170. our card
  18. Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in C++. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 129–135, Melbourne, Australia. Association fo… find
  19. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. our card
  20. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947. our card
  21. Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, and Nikolay Bogoychev. 2019. From research to production and back: Ludicrously fast neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 28… find
  22. Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations. find
  23. Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In International Conference on Learning Representations. find
  24. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations. find
  25. Hongzi Mao, Parimarjan Negi, Akshay Narayan, Hanrui Wang, Jiacheng Yang, Haonan Wang, Ryan Marcus, Mehrdad Khani Shirkoohi, Songtao He, Vikram Nathan, et al. 2019. Park: An open platform for learning-augmented computer systems. In Advances in Neural Information Processing Systems, pages 2490–2502. find
  26. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Florence, Italy. Association for Computat… our card
  27. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… our card
  28. Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. find
  29. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331–335, Florence, Italy. Association for Computational Linguistics. our card
  30. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Conference on Neural Information Processing Systems. our card
  32. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I… our card
  33. Hanrui Wang, Kuan Wang, Jiacheng Yang, Linxiao Shen, Nan Sun, Hae-Seung Lee, and Song Han. 2020a. Gcn-rl circuit designer: Transferable transistor sizing with graph neural networks and reinforcement learning. In ACM/IEEE 57th Design Automation Conference (DAC). find
  34. Hanrui Wang, Jiacheng Yang, Hae-Seung Lee, and Song Han. 2018. Learning to design circuits. In NeurIPS 2018 Machine Learning for Systems Workshop. find
  35. Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4095–4104, Stockholmsmssan, Stockholm… find
  36. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, Florence, Italy. Association for Co… our card
  37. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. our card
  38. Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. 2020b. Apq: Joint search for network architecture, pruning and quantization policy. In Conference on Computer Vision and Pattern Recognition. find
  39. David So, Quoc Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5877–5886, Long Beach, California, USA. PMLR. find
  40. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In Proceedings of… our card
  41. Bichen Wu, Xiaoliang Dai, Peizhao 4.097665 Zhang, Yanghan 27.9 Wang, Fei Sun, Yiming Wu, 4.165567 Yuandong Tian, Peter 27.9 Vajda, Yangqing Jia, and Kurt Keutzer. 2019a. 28.0 Fb-4.165906 net: Hardware-aware efficient convnet design via 4.172398 27.8 differentiable neural architecture search. In The… find
  42. Felix Wu, Angela Fan, Alexei Baevski, 4.468512 Yann Dauphin, 26.9 and Michael Auli. 2019b. Pay less attention 25.8 with 4.704716 lightweight and dynamic convolutions. In Interna-4.705631 25.8 tional Conference on Learning Representations. 25.8 4.717223 4.40 27.6 Zhanghao Wu, Zhijian Liu, Ji Lin, Yuj… find
  43. Zhongxia Yan, Hanrui Wang, Demi Guo, and Song Validation Loss From-Scratch Han. 2020. Micronet for efficient language model-BLEU ing. Journal of Machine Learning Research. 3.448210 41.8 3.454411 Su. 2018. 41.5 Biao Zhang, Deyi Xiong, and Jinsong Accel-41.7 erating neural transformer 3.454629 via an… — [arXiv:1805.00631](https://arxiv.org/abs/1805.00631) find
  44. Zhekai Zhang, Hanrui Wang, Song Han, and William J 3.477138 41.1 Dally. 2020. Sparch: Efficient architecture for 3.478086 41.1 sparse matrix multiplication. In 2020 IEEE Interna-3.483934 41.1 tional Symposium on High Performance Computer 3.487714 Architecture (HPCA), pages 261–274. IEEE. 41.0 3.5334… find
  45. Barret Zoph and Quoc V Le. 3.533737 2017. Neural Architec-40.7 ture Search with Reinforcement Inter-3.547844 Learning. In 40.6 national Conference on Learning Representations. 3.549524 40.6 3.707655 40.0 3.924879 38.8 3.926471 38.7 find
  46. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and 3.707659 40.1 Quoc V Le. 2018. Learning Transferable Architec-3.708909 tures for Scalable Image Recognition. In The 40.1 IEEE 3.749576 and Pattern Recog-39.7 Conference on Computer Vision nition (CVPR). 3.916193 39.1 *Figure 11: The validation loss… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-hat-hardware-aware-transformers-efficient/