Elamur Research
☰
home / papers / 2021 / Carbon Emissions and Large Neural Network Training

Carbon Emissions and Large Neural Network Training

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2104.10350

Summary

The paper asks how much electricity and greenhouse-gas output (CO2e) the training of large NLP models actually requires. It also asks which choices make the biggest difference. The authors, from Google and UC Berkeley, estimate training energy and emissions for T5, Meena, GShard, Switch Transformer and GPT-3. They use measured power, hardware details, datacenter efficiency (PUE) and the local energy mix. They also revisit an earlier estimate for the neural architecture search behind Evolved Transformer and find it was heavily overstated. The study covers only the operation of computers, not their manufacturing or recycling. Its main finding is that three choices compound: model architecture (sparse versus dense), processor and datacenter location. Together they can shrink the carbon footprint by two to three orders of magnitude. This also makes after-the-fact estimates unreliable. The authors argue that energy and CO2e should be reported in papers and treated as an evaluation metric. They are working with MLPerf to add energy measurement to its benchmarks.

Key points

  • Revised estimate: the CO2e estimate for the Evolved Transformer architecture search was revised downward by 88X. The new figure was based on the real hardware and datacenter used.
  • Running example: replacing Transformer (Big), run with P100 GPUs at a US datacenter of average efficiency (PUE 1.59), with Evolved Transformer (Medium), run with TPU v2 at Google's Iowa datacenter (PUE 1.11), lowered CO2e by about 57X.
  • Size of each factor:
    • Sparsely activated models can use under a tenth of the energy of dense ones.
    • Datacenter location changes CO2e by roughly 5–10X.
    • Cloud datacenters are about 1.4–2X more efficient than typical ones.
    • ML accelerators are about 2–5X more effective than off-the-shelf hardware.
  • GShard-600B: the MoE model needed about 24 MWh and 4.3 net tCO2e to train. A comparable dense model would need roughly 10X more compute while translating worse.
  • Inference share: inference reportedly makes up 80–90% of ML workload, so serving efficiency matters as much as training efficiency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

182 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.10350 · arXiv 2104.10350

@misc{patterson2021carbon,
  title = {{Carbon Emissions and Large Neural Network Training}},
  author = {Patterson, David and Gonzalez, Joseph and Le, Quoc and Liang, Chen and Munguia, Lluis-Miquel and Rothchild, Daniel and So, David and Texier, Maud and Dean, Jeff},
  year = {2021},
  doi = {10.48550/arxiv.2104.10350},
  eprint = {2104.10350},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.10350}
}

References

55 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adiwardana, D. , Luong, M., R. So, D., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., and Le. Q. Towards a Human-like Open-Domain Chatbot . arXiv preprint arXiv:2001.09977 . find our card
  2. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I. and Zaharia, M., 2010. A view of cloud computing. Communications of the ACM, 53(4), pp.50-58. find
  3. Barr, J. December 3, 2019. Amazon EC2 Update, aws.amazon.com/blogs/aws/amazon-ec2-update-inf1-instances-with-aws-inferentia-chips -for-high-performance-cost-effective-inferencing/ . find
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam , P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess,… find
  5. Bender, E., Gebru, T., McMillan-Major, A. Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? FAccT 2021. http://faculty.washington.edu/ebender/papers/Stochastic\_Parrots.pdf . ↗
  6. Carbon Offset Research and Education, 2021, Carbon Offset Guide, https://www.offsetguide.org/ . ↗
  7. Chang, K.W., Prabhakaran, V. and Ordonez, V., 2019, November. Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial… find our card
  8. Criddle, C., February 10, 2021. Bitcoin consumes more electricity than Argentina, www.bbc.com/news/technology-56012952 . find
  9. Digiconomist, 2021, Bitcoin Energy Consumption Index, https://digiconomist.net/bitcoin-energy-consumption/ . ↗
  10. Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N., 2019. Show Your Work: Improved Reporting of Experimental Results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMN… find
  11. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H. and Smith, N., 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 . find
  12. Apache-licensed Evolved Transformer open-source implementation in tensorflow/tensor2tensor GitHub repository. https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/evolved\_transformer.py ↗
  13. Fedus, W., Zoph, B., Shazeer, N., January 11, 2021, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://arxiv.org/abs/2101.03961 . ↗
  14. Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E.H. and Beutel, A., 2019, January. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 219-226). https://research.google/pubs/pub47670/ . ↗
  15. Google, December 2016, Achieving Our 100% Renewable Energy Purchasing Goal and Going Beyond, https://static. googleusercontent.com/media/www.google.com/en//green/pdf/achieving-100-renewable-energy-purchasing-goal .pdf . find
  16. Google, Environmental Report 2020, https://www.gstatic.com/gumdrop/sustainability/google-2020-environmental-report.pdf . ↗
  17. Google, February 2021, 24/7 Carbon-Free Energy: Methodologies and Metrics, https://www.gstatic.com/gumdrop/sustainability/24x7-carbon-free-energy-methodologies-metrics.pdf . ↗
  18. Gupta, U., Kim, Y.G., Lee, S., Tse, J., Lee, H.H.S., Wei, G.Y., Brooks, D. and Wu, C.J., 2020. Chasing Carbon: The Elusive Environmental Footprint of Computing. arXiv preprint arXiv:2011.02839 . — [arXiv:2011.02839](https://arxiv.org/abs/2011.02839) find
  19. Hall, K., May 4, 2020, An NLU-Powered Tool to Explore COVID-19, https://ai.googleblog.com/2020/05/an-nlu-powered-tool-to-explore-covid-19.html . ↗
  20. Han, S., Pool, J., Tran, J. and Dally, W.J., 2015. Learning both weights and connections for efficient neural networks. ICLR 2016. arXiv preprint arXiv:1510.00149 . find
  21. Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D. and Pineau, J., 2020. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research. https://jmlr.org/papers/v21/20-312.html ↗
  22. Hernandez, D. and Brown, T.B., 2020. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04305. https://arxiv.org/abs/2005.04305 . — [arXiv:2005.04305](https://arxiv.org/abs/2005.04305) ↗
  23. Hinton, G., Vinyals, O. and Dean, J., 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 . find our card
  24. Hölzle, U., Feb 27, 2020. datacenters are more energy efficient than ever. blog.google/outreach-initiatives/sustainability/data-centers-energy-efficient find
  25. Johnson, M., April 22, 2020, A Scalable Approach to Reducing Gender Bias in Google Translate, https://ai.googleblog.com/2020/04/a-scalable-approach-to-reducing-gender.html . ↗
  26. Jouppi, N., Yoon, D-H, Jablin, T., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., Patil, N.,Prasad, S., Young, C., Zhou, Z., and Patterson, D., May 2021. Ten Lessons From Three Generations Shaped Google’s TPUv4i, to appear, the 48th International Symposium on Computer Architecture. find
  27. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. find
  28. Kärcher B. Formation and radiative forcing of contrail cirrus. Nature communication s. 2018 May 8;9(1):1-7. https://www.nature.com/articles/s41467-018-04068-0 . ↗
  29. Kuczmarski, J. and Johnson, M., 2018. Gender-aware natural language translation. www.tdcommons.org/dpubs\_series/1577/ . find
  30. Lacoste, A., Luccioni, A., Schmidt, V. and Dandres, T., 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 . find our card
  31. Lannelongue, L., Grealey, J. and Inouye, M., 2020. Green algorithms: Quantifying the carbon footprint of computation. arXiv: 2007.07610 . find
  32. Leopold, G. March 19, 2019, AWS to Offer Nvidia’s T4 GPUs for AI Inferencing, www.hpcwire.com/2019/03/19/aws-upgrades-its-gpu-backed-ai-inference-platform/ . find
  33. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N. and Chen, Z., 2020. GShard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 . find
  34. Li, S., Tan, M., Pang, R., Li, A., Cheng, L., Le, Q. and Jouppi, N.P., 2021. Searching for Fast Model Families on Datacenter Accelerators. arXiv preprint arXiv:2102.05610 . — [arXiv:2102.05610](https://arxiv.org/abs/2102.05610) find
  35. Liu, H., Simonyan, K. and Yang, Y., 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 . — [arXiv:1806.09055](https://arxiv.org/abs/1806.09055) find
  36. Luccioni, A., and Schmidt, V.. March 2021, Private Communication. find
  37. Masanet, E., Shehabi, A., Lei, N., Smith, S. and Koomey, J., 2020. Recalibrating global datacenter energy-use estimates. Science , 367(6481), pp.984-986. https://datacenters.lbl.gov/sites/default/files/Masanet\_et\_al\_Science\_2020.full\_.pdf . ↗
  38. Masanet, E., March 24, 2021, Data Center Energy Analysis: Past, Present, and Future , lecture at UCSB. find
  39. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. and Galstyan, A., 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635. https://arxiv.org/pdf/1908.09635.pdf . ↗ our card
  40. Pham, H., Guan, M., Zoph, B., Le, Q. and Dean, J., 2018, July. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning (pp. 4095-4104). PMLR. arXiv preprint arXiv:1802.03268 . find
  41. Radovanovic, A. April 22, 2020, Our datacenters now work harder when the sun shines and wind blows, https://blog.google/inside-google/infrastructure/data-centers-work-harder-sun-shines-wind-blows ↗
  42. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P.J., 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 . find
  43. Ritchie, H., October 22, 2020, Climate change and flying: what share of global CO2 emissions come from aviation? https://ourworldindata.org/co2-emissions-from-aviation . ↗
  44. Ryor, J.N. and Tawney, L.E.T.H.A., 2014. Utility-Scale Renewable Energy: Understanding Cost Parity. Paris: World Resources Institute. https://www.ctc-n.org/sites/www.ctc-n.org/files/resources/wri14\_factsheets\_utility\_scale\_v4.pdf . ↗
  45. Sanh, V., Debut, L., Chaumond, J. and Wolf, T., 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 . find
  46. Schwartz, R., Dodge, J., Smith, N.A. and Etzioni, O., 2020. Green AI. Communications of the ACM , 63(12), pp.54-63. https://cacm.acm.org/magazines/2020/12/248800-green-ai/fulltext . ↗ our card
  47. Schleier-Smith, J., Sreekanti, V., Khandelwal, A., Carreira, J., Yadwadkar, N., Popa, R., Joseph E. Gonzalez,J., Ion Stoica, I., and David A. Patterson, D., 2021 What Serverless Computing Is and Should Become: The Next Phase of Cloud Computing, Communications of the ACM, 64(5) . find
  48. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G. and Dean, J., 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017. arXiv preprint arXiv:1701.06538 . find
  49. So, D., Le, Q. and Liang, C., 2019, May. The Evolved Transformer. In International Conference on Machine Learning 2019 (pp. 5877-5886). PMLR. arXiv preprint arXiv:1901.11117 . — [arXiv:1901.11117](https://arxiv.org/abs/1901.11117) find
  50. Strubell, E., Ganesh, A. and McCallum, A., 2019. Energy and policy considerations for deep learning in NLP. ACL 2019. arXiv preprint arXiv:1906.02243 . find our card
  51. Sutskever, I. Personal Communication, February 4, 2021. find
  52. Tan, M. and Le, Q., 2019, May. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (pp. 6105-6114). PMLR. arXiv preprint arXiv:1905.11946 . find our card
  53. US Energy Information Administration, 2021, FAQ How much carbon dioxide is produced per kilowatt hour of U.S. electricity generation? https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 . ↗
  54. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I., 2017. Attention is all you need. NeurIPS 2017. arXiv preprint arXiv:1706.03762 . find
  55. Wang, Y., Yao, Q., Kwok, J.T. and Ni, L.M., 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys , 53(3), pp.1-34. find