Elamur Research
☰
home / papers / 2019 / Quantifying the Carbon Emissions of Machine Learning

Quantifying the Carbon Emissions of Machine Learning

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1910.09700

Summary

Training neural networks consumes considerable energy, but ML practitioners rarely know how much CO2 their experiments produce. The authors built the Machine Learning Emissions Calculator, a web tool hosted at mlco2.github.io/impact. A user enters three things: the server's region, the GPU model, and how long training ran. The tool returns an approximate figure in CO2-equivalents (CO2eq).

Behind the tool is a public dataset. It brings together hardware power draw, the regions where Amazon Web Services, Microsoft Azure and Google Cloud Platform offer GPUs, the carbon intensity of each local grid, and any offsets a provider purchases. The authors assume each data center draws power from its local grid.

The central finding is that location dominates. Depending on the grid, the same job can emit up to about 40 times more carbon. Hardware choice and wasteful experimental practices also matter.

The paper offers practical advice and acknowledges its limits: aggregate load balancing, opaque provider data, and inference costs are not covered. The tool gives researchers a simple, editable way to estimate their footprint and to treat it as a design decision.

Key points

  • Tool and data: The calculator's open data lives in the mlco2/impact GitHub repository. Anyone can correct it through issues and pull requests. The site also includes educational pages on concepts such as RECs and carbon neutrality.
  • Effect of grid location:
    • Within North America, carbon intensity runs from about 20 g CO2eq/kWh in Quebec to 736.6 g in Iowa.
    • Worldwide, it spans roughly 20 to 820 g, a gap of about 40×.
    • For multi-week runs of models like VGG or BERT, choosing a hydro-powered site can avoid hundreds of kilograms of CO2eq.
  • Hardware efficiency (FLOPS/W, based on TDP):
    • CPUs can be around 10× less efficient than GPUs.
    • TPUs can be 4–8× more efficient than GPUs.
    • The embedded Jetson AGX Xavier can be 10–20× more efficient than conventional GPUs.
  • Provider factors:
    • Google Cloud reports an average PUE of 1.1, so about 11% of its energy goes to overhead such as cooling.
    • Google and Azure are certified carbon neutral, the latter with 44% direct renewable supply as of 2016.
    • AWS is not yet fully carbon neutral.
  • Reducing waste:
    • Prefer fine-tuning pre-trained models over training from scratch.
    • Use random search instead of grid search for hyperparameters.
    • Prevent failed or uninformative runs through testing, early debugging and literature review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

199 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 192 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1910.09700 · arXiv 1910.09700

@misc{lacoste2019quantifying,
  title = {{Quantifying the Carbon Emissions of Machine Learning}},
  author = {Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
  year = {2019},
  doi = {10.48550/arxiv.1910.09700},
  eprint = {1910.09700},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.09700}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019. find our card
  2. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. arXiv preprint arXiv:1907.10597, 2019. find our card
  3. Simon Eggleston, Leandro Buendia, Kyoko Miwa, Todd Ngara, and Kiyoto Tanabe. 2006 IPCC guidelines for national greenhouse gas inventories, volume 5. Institute for Global Environmental Strategies Hayama, Japan, 2006. find
  4. WM To and Peter KC Lee. Ghg emissions from electricity consumption: A case study of hong kong from 2002 to 2015 and trends to 2030. Journal of cleaner production, 165:589–598, 2017. find
  5. Matthew Brander, Aman Sood, Charlotte Wylie, Amy Haughton, and Jessica Lovell. Electricity-specific emission factors for grid electricity. Ecometrica, Emissionfactors. com, 2011. find
  6. Ari Harju, Topi Siro, Filippo Federici Canova, Samuli Hakala, and Teemu Rantalaiho. Computational physics on graphics processing units. In Proceedings of the 11th international conference on Applied Parallel and Scientific Computing, pages 3–26. Springer-Verlag, 2012. find
  7. Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher B Kendall, Michael B Gotway, and Jianming Liang. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging, 35(5):1299–1312, 2016. find
  8. Keiji Yanai and Yoshiyuki Kawano. Food image recognition using deep convolutional network with pre-training and fine-tuning. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1–6. IEEE, 2015. find
  9. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. find
  10. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281–305, 2012. find
  11. Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.01774, 2018. — [arXiv:1807.01774](https://arxiv.org/abs/1807.01774) find
  12. Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning, pages 3–33. Springer, 2019. find
  13. Google. Google environmental report 2018, 2018. find
  14. Microsoft. Beyond carbon neutral. white paper, 2018. find
  15. Amazon Web Services. Aws & sustainability, 2019. find
  16. Jim Gao. Machine learning applications for data center optimization, 2014. find
  17. Google Data Centers efficiency: How we do it. https://www.google.com/about/ datacenters/efficiency/internal/, 2019. Accessed: 2019-08-23. ↗
  18. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. find our card
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  20. Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560, 2016. — [arXiv:1603.06560](https://arxiv.org/abs/1603.06560) find
  21. Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05934, 2018. — [arXiv:1810.05934](https://arxiv.org/abs/1810.05934) find
  22. Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.01774, 2018. — [arXiv:1807.01774](https://arxiv.org/abs/1807.01774) find
  23. Paul Teich. Tearing apart google’s tpu 3.0 ai coprocessor. https://www.nextplatform. com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/, 2018. ↗
  24. Ermao Cai, Da-Cheng Juan, Dimitrios Stamoulis, and Diana Marculescu. Neuralpower: Predict and deploy energy-efficient convolutional neural networks. arXiv preprint arXiv:1710.05420, 2017. find
  25. David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. Tackling climate change with machine learning. arXiv preprint arXiv:1906.05433, 2019. — [arXiv:1906.05433](https://arxiv.org/abs/1906.05433) find
  26. Victor Schmidt, Alexandra Luccioni, S. Karthik Mukkavilli, Narmada Balasooriya, Kris Sankaran, Jennifer Chayes, and Yoshua Bengio. Visualizing the consequences of climate change using cycle-consistent adversarial networks. CoRR, abs/1905.03709, 2019. — [arXiv:1905.03709](https://arxiv.org/abs/1905.03709) find