Elamur Research⌕
☰
home / papers / 2022 / A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

Paper · 2022 · Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol et al.

Summary

Papers on large language models usually report only the electricity used for the final training runs. This one asks what the whole project costs in carbon. The case study is Noor, a joint effort by TII (UAE) and LightOn (France) to build Arabic generative models larger than AraGPT. The suite has four models (1.5B, 2.7B, 6.7B and 13B parameters) trained on a curated corpus of about 150B tokens modelled on The Pile.

The authors count emissions from:

  • collecting, filtering and storing the data;
  • exploratory R&D runs;
  • main pretraining;
  • projected serving;
  • staff-related items such as international flights.

They first lay out the levers that shape a model's footprint: model size, hardware efficiency, data-center PUE, grid carbon intensity and modelling choices. The project total comes to roughly 36.5 tCO2e. Pretraining dominates, but flights, storage and experiments together make up about a third. Inference could later rival training. Placing all compute on a low-carbon grid would cut the total by more than half. The paper argues for routine full-lifecycle carbon reporting.

Key points

  • The total footprint is estimated at 36.5 tCO2e. About 65% comes from training, 18% from flights, 12% from data storage and 4% from small R&D runs. That is a little over two years of an average US resident's emissions.
  • Running everything in France would have cut the total to 14.9 tCO2e. In that scenario flights become about 42% of what remains.
  • Training compute is approximated as C = 6ND (Kaplan et al.), and per-token inference costs about a third of a training step.
  • Grid carbon intensity varies widely: the world average is about 475 gCO2e/kWh, while Google's Montreal site reports about 27. A PUE of 1.1 instead of the 1.8 world average saves about 39% of energy.
  • Pipeline details:
    • The data was filtered with an LM-based quality scorer in the style of CCNet.
    • R&D models ranged from 100M to 1.5B parameters, trained on 10–30B tokens.
    • Hardware manufacturing (lifecycle) emissions are explicitly excluded because the data is unavailable.

Interesting ideas

  • I1. Treat the carbon bill as a project-wide ledger split into cost centers (data, experiments, main runs, serving, travel), not one training figure; this exposes hidden costs that dominate once the main compute is clean.
  • I2. Report a counterfactual scenario next to the real one (here, the same project on a clean grid) to show which single decision moves the total most.
  • I3. Project serving costs ahead of time from expected token volume, since cumulative inference can overtake one-off training.
  • I4. Tokenizer fertility affects efficiency directly: fewer tokens for the same text means less compute everywhere downstream.
  • I5. Score corpus quality with a language model before training, CCNet-style, so compute is not wasted on junk text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Track compute per pipeline stage (witness parsers, LLM teacher calls, retraining, self-play) so we can see which part of silver-data production actually costs the most.
  • I2 (learn): Evaluation reports could pair each result with a "what if" baseline, e.g. accuracy with versus without the LLM teacher tier, to justify each expensive component.
  • I3 (answer): Estimate cost per answered question across the full text → tree → frames → world chain, to decide where caching compiled world knowledge pays off.
  • I4 (tokens): Measure tokens-per-sentence for the en crate's tokenizer on our treebanks, since tighter segmentation would speed up gparse and every later stage.
  • I5 (learn): Run a cheap quality filter on raw text before sending it to the multi-parser silver pipeline, so agreement voting is spent only on sentences worth annotating.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Technology Innovation InstituteAbu Dhabi · United Arab Emirates7 works · 543 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksHardware and Architecture · 81 worksEngineering · 133 worksElectrical and Electronic Engineering · 18 worksEnvironmental Science · 9 worksEnvironmental Engineering · 3 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah and Julien Launay. 2022. A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model. Proceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.bigscience-1.8

@inproceedings{lakim2022holistic,
  title = {{A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model}},
  author = {Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah and Julien Launay},
  booktitle = {Proceedings of BigScience Episode \#5 – Workshop on Challenges \& Perspectives in Creating Large Language Models},
  year = {2022},
  doi = {10.18653/v1/2022.bigscience-1.8},
  url = {https://doi.org/10.18653/v1/2022.bigscience-1.8}
}

References

43 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021. Glam: Efficient scaling of language models with mixture-of-experts.
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Aragpt2: Pre-trained transformer for arabic language generation. ↗
  3. Joshua Aslan, Kieren Mayers, Jonathan Koomey, and Chris France. 2017. Electricity intensity of internet data transmission: Untangling the estimates: Electricity intensity of data transmission. Journal of Industrial Ecology, 22. ↗
  4. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. ↗
  5. Clarisse Aujoux, Kumiko Kotera, and Odile Blanchard. 2021. Estimating the carbon footprint of the grand project, a multi-decade astrophysics experiment. Astroparticle Physics, 131:102587. ↗
  6. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. ↗
  7. Jayant Baliga, Robert Ayre, Kerry Hinton, and Rodney Tucker. 2011. Green cloud computing: Balancing energy in processing, storage, and transport. Proceedings of the IEEE, 99:149 – 167. ↗
  8. Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. 2021. Chasing carbon: The elusive environmental footprint of computing. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 854–867. IEEE. ↗
  9. Nesrine Bannour, Sahar Ghannay, Aurélie Névéol, and Anne-Laure Ligozat. 2021. Evaluating the carbon footprint of nlp methods: a survey and analysis of existing tools. In EMNLP, Workshop SustaiNLP. find
  10. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. ↗
  11. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701. ↗
  12. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ↗
  13. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. ↗
  14. Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. 2021. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. arXiv preprint arXiv:2109.04650. ↗
  15. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ↗
  16. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700. ↗
  17. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs. find
  18. Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. 2022. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01169. ↗
  19. Kadan Lottick, Silvia Susai, Sorelle A. Friedler, and Jonathan P. Wilson. 2019. Energy usage reports: Environmental awareness as part of algorithmic accountability. Workshop on Tackling Climate Change with Machine Learning at NeurIPS 2019. ↗
  20. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Valérie Masson-Delmotte, Panmao Zhai, Hans-Otto Pörtner, Debra Roberts, Jim Skea, Priyadarshi R Shukla, Anna Pirani, W Moufouma-Okia, C Péan, Matthew Skiles, Euijin Yang, Orad Reshef, Diego Robalino Muñoz, Diana Cintron, Mary Laura Lind, Alexander Rush, Patricia Perez Calleja, Robert Nerenberg, Andrea Armani, et al. 2021. Conference demographics and footprint changed by virtual platforms. Nature Sustainability, pages 1–8. find
  21. R Pidcock, et al. 2018. Global warming of 1.5 c. An IPCC Special Report on the impacts of global warming of, 1(5). find
  22. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15. find
  23. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990. ↗
  24. David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Hung Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeffrey Dean. 2022. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink. ↗
  25. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in nlp.
  26. Mariarosaria Taddeo, Andreas Tsamados, Josh Cowls, and Luciano Floridi. 2021. Artificial intelligence and the climate emergency: Opportunities, challenges, and recommendations. One Earth, 4:776– 779. ↗
  27. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. ↗
  28. Ashley Pilipiszyn. 2021. Gpt-3 powers the next generation of apps. find
  29. Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. 2020. The computational limits of deep learning. find
  30. Lorenzo Posani, Alessio Paccoia, and Marco Moschet-tini. 2019. The carbon footprint of distributed cloud storage. ↗
  31. Nenad Tomašev, Julien Cornebise, Frank Hutter, Shakir Mohamed, Angela Picciariello, Bec Connelly, Danielle Belgrave, Daphne Ezer, Fanny Cachat van der Haert, Frank Mugisha, et al. 2020. Ai for social good: unlocking the opportunity for positive impact. Nature Communications, 11(1):1– 6. ↗
  32. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. ↗
  33. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR. ↗
  34. Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. 2020. Survey of machine learning accelerators. In 2020 IEEE high performance extreme computing conference (HPEC), pages 1–12. IEEE. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  36. Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. 2021. Ai accelerator survey and trends. In 2021 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9. IEEE. find
  37. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax . ↗
  38. Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. 2021. CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing. find
  39. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data.
  40. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63(12):54–63.
  41. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. 2022. Compute trends across three eras of machine learning. arXiv preprint arXiv:2202.05924. ↗
  42. David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. 2022. Sustainable ai: Environmental implications, challenges and opportunities. find
  43. Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xiansheng Hua. 2019. Quantization networks. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lakim-2022-holistic-assessment-carbon-footprint-noor/