Elamur Research
☰
home / papers / 2019 / Green AI

Green AI

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1907.10597

Summary

This position paper looks at the rapidly growing computational demands of deep learning research. Citing an estimate that the compute used to train landmark models grew roughly 300,000-fold between 2012 and 2018, the authors argue that chasing leaderboard accuracy by spending ever more compute has environmental costs (carbon emissions). It also has social costs, because researchers without large budgets are shut out, especially in academia and emerging economies. They call this compute-heavy style "Red AI" and break the cost of producing a result into three multiplying factors: cost per example, training-set size, and the number of hyperparameter or architecture experiments. To show how little attention efficiency gets, they surveyed 60 papers from ACL, NeurIPS and CVPR. The proposed alternative, "Green AI", treats efficiency as a reviewable contribution alongside accuracy. In practice this means reporting a cost "price tag" for each result, plotting performance against compute budget and training data size, and continuing to release pretrained models publicly. The paper matters because it gave the community a shared vocabulary and practical reporting habits for weighing accuracy gains against resource use.

Key points

  • Cost formula: the cost of a result scales with E·D·H, where E is the cost of processing one example, D is the dataset size and H is the number of hyperparameter trials.
  • Survey of 60 papers: accuracy was the main goal in 90% of ACL, 80% of NeurIPS and 75% of CVPR papers. Efficiency was the main goal in only 10% of ACL and 20% of CVPR papers.
  • Scale examples: BERT-large (about 350M parameters, 64 TPUs for 4 days); Grover and openGPT2-XL (1.5B parameters each, with Grover costing about $25,000 to train); XLNet (512 TPUs, 32B word-pieces); RoBERTa (160GB of text, about 25,000 GPU hours).
  • Experiment counts: one neural architecture search trained more than 12,800 networks, and one LSTM study tried 1,500 hyperparameter settings. The June 2019 Common Crawl snapshot alone is 242 TB.
  • Recommendations: use a standard, hardware-independent efficiency measure (e.g., operation counts), report expected validation performance as a function of budget, and report results at several pretraining data sizes. The authors note this last point helps low-resource languages in particular.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

297 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 283 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.10597 · arXiv 1907.10597

@misc{schwartz2019green,
  title = {{Green AI}},
  author = {Schwartz, Roy and Dodge, Jesse and Smith, Noah A. and Etzioni, Oren},
  year = {2019},
  doi = {10.48550/arxiv.1907.10597},
  eprint = {1907.10597},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.10597}
}

References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prabal Acharyya, Sean D Rosario, Roey Flor, Ritvik Joshi, Dian Li, Roberto Linares, and Hongbao Zhang. Autopilot of cement plants for reduction of fuel consumption and emissions, 2019. ICML Workshop on Climate Change. find
  2. Dario Amodei and Danny Hernandez. AI and compute, 2018. Blog post. find
  3. James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Proc. of NeurIPS, 2011. find
  4. Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. In Proc. of ISCAS, 2017. find
  5. Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Proc. of NeurIPS, 2017. find
  6. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. of CVPR, 2009. find
  7. Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance, 2019. arXiv:1907.04840. — [arXiv:1907.04840](https://arxiv.org/abs/1907.04840) find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, 2019. find
  9. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proc. of EMNLP, 2019. find
  10. Jesse Dodge, Kevin Jamieson, and Noah A. Smith. Open loop hyperparameter optimization and determinantal point processes. In Proc. of AutoML, 2017. find
  11. Clement Duhart, Gershon Dublon, Brian Mayton, Glorianna Davenport, and Joseph A. Paradiso. Deep learning for wildlife conservation and restoration efforts, 2019. ICML Workshop on Climate Change. find
  12. Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. MorphNet: Fast & simple resource-constrained structure learning of deep networks. In Proc. of CVPR, 2018. find
  13. Alon Halevy, Peter Norvig, and Fernando Pereira. The unreasonable effectiveness of data. IEEE Intelligent Systems, 24:8–12, 2009. find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of CVPR, 2016. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  16. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications, 2017. arXiv:1704.04861. find our card
  17. Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proc. of CVPR, 2018. find
  18. Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. In Proc. of CVPR, 2017. find
  19. Sanket Kamthe and Marc Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In Proc. of AISTATS, 2018. find
  20. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NeurIPS, 2012. find
  21. Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In Proc. of ICLR, 2017. find
  22. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Proc. of ECCV, 2016. find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach, 2019. arXiv:1907.11692. find
  24. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In Proc. of ECCV, 2018. find
  25. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proc. ECCV, 2018. find
  26. Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In Proc. of EMNLP, 2018. find
  27. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In Proc. of ICLR, 2017. find
  28. Gordon E. Moore. Cramming more components onto integrated circuits, 1965. find
  29. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018. find
  30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. OpenAI Blog. find
  31. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proc. of ECCV, 2016. find
  32. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proc. of CVPR, 2016. find
  33. David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, Alexandra Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Körding, Carla Gomes, Andrew Y. Ng,… find our card
  34. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proc. of CVPR, 2018. find
  35. Roy Schwartz, Sam Thomson, and Noah A. Smith. SoPa: Bridging CNNs, RNNs, and weighted finite-state machines. In Proc. of ACL, 2018. find
  36. Yoav Shoham, Raymond Perrault, Erik Brynjolfsson, Jack Clark, James Manyika, Juan Carlos Niebles, Terah Lyons, John Etchemendy, and Z Bauer. The AI index 2018 annual report. AI Index Steering Committee, Human-Centered AI Initiative, Stanford University. Available at http://cdn.aiindex.org/2018/AI% 2… ↗
  37. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcu… find
  38. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm… find our card
  39. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go withou… find
  40. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proc. of ACL, 2019. find our card
  41. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proc. of ICCV, 2017. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS, 2017. find
  43. Tom Veniat and Ludovic Denoyer. Learning time/memory-efficient deep architectures with budgeted super networks. In Proc. of CVPR, 2018. find
  44. Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early fusion for goal directed robotic vision. In Proc. of IROS, 2019. find
  45. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems, 2019. arXiv:1905.00537. find our card
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR, 2019. find our card
  47. Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. of CVPR, 2017. find
  48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding, 2019. arXiv:1906.08237. find our card
  49. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news, 2019. arXiv:1905.12616. find our card
  50. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proc. of CVPR, 2018. find
  51. Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Proc. of ICLR, 2017. find our card