Elamur Research
☰
home / papers / 2019 / Energy and Policy Considerations for Deep Learning in NLP

Energy and Policy Considerations for Deep Learning in NLP

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1355.pdf

Summary

Large neural NLP models now get the best accuracy, but training them takes a great deal of compute. The authors argue that this has real financial and climate costs and limits who can do the research. To measure this, they ran popular open-source models (Transformer, ELMo, BERT, GPT-2) on their own GPUs for up to one day each. They also covered the evolved-Transformer neural architecture search (NAS) of So et al. GPU power was logged with nvidia-smi and CPU/DRAM power with Intel RAPL. They scaled these readings up to the full training times reported in the original papers and applied a data-centre PUE of 1.58. Energy was converted to CO2e using the U.S. EPA average of 0.954 lbs/kWh, and each run was priced on cloud GPUs and TPUs. A case study of their own LISA pipeline (parsing, SRL) adds the cost of tuning and full R&D. Training one model produces tens to about 1,400 lbs CO2e, while the NAS run produces about five times a car's lifetime emissions. The paper made energy reporting and fair access to compute mainstream concerns in NLP.

Key points

  • BERT base on 64 V100 GPUs (about 79 hours) is estimated at roughly 1,438 lbs CO2e. That is close to one passenger's New York–San Francisco round trip by air (about 1,984 lbs).
  • NAS for the Transformer big model needs about 274,120 P100-hours. That works out to roughly 626,155 lbs CO2e and $0.94M–$3.2M in cloud costs, compared with 126,000 lbs for an average car's lifetime including fuel.
  • For workloads suited to them, TPUs cost less than GPUs. Training BERT on 16 TPUv2 chips costs about $2.1k–$6.9k, versus $3.8k–$12.6k on 64 V100s.
  • Developing LISA kept about 60 GPUs busy for six months, for an estimated 78,468 lbs CO2e. Retuning it for a new dataset was estimated at 24 jobs.
  • The authors make three recommendations:
    • Report training time and how sensitive results are to hyperparameters, using hardware-independent measures such as gigaflops to convergence.
    • Fund shared, government-backed academic compute. Buying the case-study hardware would cost about $145k, roughly half the on-demand cloud price.
    • Prioritise efficient algorithms and hardware, and make Bayesian or random hyperparameter search easier to use than brute-force grid search.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

596 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 567 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1355 · arXiv 1906.02243

@inproceedings{strubell2019energy,
  title = {{Energy and Policy Considerations for Deep Learning in NLP}},
  author = {Strubell, Emma and Ganesh, Ananya and McCallum, Andrew},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1355},
  eprint = {1906.02243},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1355}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rhonda Ascierto. 2018. Uptime Institute Global Data Center Survey. Technical report, Uptime Institute. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference for Learning Representations (ICLR), San Diego, California, USA. find our card
  3. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. find
  4. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  5. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281–305. find
  6. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951–2959. find
  7. James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pages 2546–2554. find
  8. David R. So, Chen Liang, and Quoc V. Le. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning (ICML). find
  9. Bruno Burger. 2019. Net Public Electricity Generation in Germany in 2018. Technical report, Fraunhofer Institute for Solar Energy Systems ISE. find
  10. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Semantic Role Labeling. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium. find our card
  11. Alfredo Canziani, Adam Paszke, and Eugenio Culur-ciello. 2016. An analysis of deep neural network models for practical applications. find
  12. Gary Cook, Jude Lee, Tamina Tsai, Ada Kongn, John Deans, Brian Johnson, Elizabeth Jardim, and Brian Johnson. 2017. Clicking Clean: Who is winning the race to build a green internet? Technical report, Greenpeace. find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS). find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL. find
  15. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR. find our card
  16. EPA. 2018. Emissions & Generation Resource Integrated Database (eGRID). Technical report, U.S. Environmental Protection Agency. find
  17. Christopher Forster, Thor Johnsen, Swetha Man-dava, Sharath Turuvekere Sreenivas, Deyu Fu, Julie Bernauer, Allison Gray, Sharan Chetlur, and Raul Puri. 2019. BERT Meets GPUs. Technical report, NVIDIA AI. find
  18. Da Li, Xinbo Chen, Michela Becchi, and Ziliang Zong. 2016. Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus. 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Co… find
  19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. Association for Computational Linguistics. find