Elamur Research⌕
☰
home / papers / 2024 / Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Paper · 2024 · Chris Cummins, Volker Seeker, Dejan Grubisic et al.

Summary

General-purpose code LLMs see very little assembly or compiler IR during training, so they handle optimization poorly and often produce broken code. Training a specialised model from scratch is too expensive for most researchers. The authors address this by releasing LLM Compiler, a family of open foundation models built on Code Llama. Training happened in stages. The models were first further pretrained on LLVM-IR and assembly produced with LLVM 17.0.6. They were then instruction-tuned on a "compiler emulation" task: given unoptimized IR and a random sequence of opt passes, the model predicts the transformed code and its resulting size. Two downstream variants, LLM Compiler FTD, were then fine-tuned for choosing code-size flags and for lifting x86_64/ARM assembly back to LLVM-IR. On these tasks the FTD models capture most of the gains of an autotuning search without extra compilations. They also beat Code Llama and GPT-4 Turbo. The work matters because it gives academia and industry a ready-made, commercially reusable base for compiler-focused ML.

Key points

  • Sizes are 7B and 13B parameters. The models were trained on 546B tokens: 401B for IR/assembly pretraining and 145B for compiler emulation. FTD adds 164B more (84B flag tuning, 80B disassembly), for 710B in total.
  • The pretraining data splits roughly evenly between LLVM-IR and assembly. Most of it targets x86_64, with some aarch64 and a small amount of CUDA (nvptx64).
  • The compiler emulation data applies random lists of 1–50 passes. The passes are drawn from 167 of the 346 available opt arguments.
  • LLM Compiler FTD reaches 77% of the code-size potential of autotuning. Its disassembly round-trips correctly in 45% of cases, with 14% exact matches.
  • The main limitation is the 16k-token context window. About 67% of MiBench translation units do not fit as flag-tuning prompts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksHardware and Architecture · 183 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve and Hugh Leather. 2024. Meta Large Language Model Compiler: Foundation Models of Compiler Optimization. arXiv:2407.02524.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.02524 · arXiv 2407.02524

@misc{cummins2024meta,
  title = {{Meta Large Language Model Compiler: Foundation Models of Compiler Optimization}},
  author = {Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve and Hugh Leather},
  year = {2024},
  doi = {10.48550/arxiv.2407.02524},
  eprint = {2407.02524},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.02524}
}

References

61 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. F. Agakov, E. Bonilla, J. Cavazos, B. Franke, G. Fursin, M.F.P. O’Boyle, J. Thomson, M. Toussaint, and C.K.I. Williams. Using machine learning to focus iterative optimization. In CGO, 2006. find
  2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Rom… — SantaCoder: don't reach for the stars!
  3. Jordi Armengol-Estapé and Michael FP O’Boyle. Learning C to x86 Translation: An Experiment in Neural Compilation. arXiv:2108.07639, 2021. — [arXiv:2108.07639](https://arxiv.org/abs/2108.07639) find
  4. Jordi Armengol-Estapé, Jackson Woodruff, Chris Cummins, and Michael FP O’Boyle. SLaDe: A Portable Small Language Model Decompiler for Optimized Assembler. In CGO, 2024. find
  5. Amir H Ashouri, Mostafa Elhoushi, Yuzhe Hua, Xiang Wang, Muhammad Asif Manzoor, Bryan Chan, and Yaoqing Gao. MLGOPerf: An ML Guided Inliner to Optimize Performance. arXiv:2207.08389, 2022. — [arXiv:2207.08389](https://arxiv.org/abs/2207.08389) find
  6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program Synthesis with Large Language Models. arXiv:2108.07732, 2021.
  7. Ying Cao, Ruigang Liang, Kai Chen, and Peiwei Hu. Boosting Neural Networks to Decompile Optimized Binaries. In ACSAC, 2022. find
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  9. Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. End-to-End Deep Learning of Optimization Heuristics. In PACT, 2017. find
  10. Chris Cummins, Zacharias Fisches, Tal Ben-Nun, Torsten Hoefler, Michael O’Boyle, and Hugh Leather. ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. In ICML, 2021. find
  11. Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and Hugh Leather. CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research. In CGO, 2022. find
  12. Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. In ISSTA, 2023. find
  13. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. LongNet: Scaling Transformers to 1,000,000,000 Tokens. arXiv:2307.02486, 2023.
  14. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A Pre-trained Model for Programming and Natural Languages. arXiv:2002.08155, 2020.
  15. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999, 2023. find
  16. G. G. Fursin, M. F. P. O’Boyle, and P. M. W. Knijnenburg. Evaluating Iterative Compilation. In LCPC, 2005. find
  17. Philip Gage. A New Algorithm for Data Compression. C Users Journal, 12(2), 1994. find
  18. Shannon K Gallagher, William E Klieber, and David Svoboda. LLVM Intermediate Representation for Code Weakness Identification, 2022. find
  19. Aiden Grossman, Ludger Paehler, Konstantinos Parasyris, Tal Ben-Nun, Jacob Hegna, William Moses, Jose M Monsalve Diaz, Mircea Trofin, and Johannes Doerfert. ComPile: A Large IR Dataset from Production Sources. arXiv:2309.15432, 2024. — [arXiv:2309.15432](https://arxiv.org/abs/2309.15432) find
  20. Dejan Grubisic, Chris Cummins, Volker Seeker, and Hugh Leather. Compiler generated feedback for Large Language Models. arXiv:2403.14714, 2024a. — [arXiv:2403.14714](https://arxiv.org/abs/2403.14714) find
  21. Dejan Grubisic, Chris Cummins, Volker Seeker, and Hugh Leather. Priority Sampling of Large Language Models for Compilers. arXiv:2402.18734, 2024b. — [arXiv:2402.18734](https://arxiv.org/abs/2402.18734) find
  22. Yi Gui, Yao Wan, Hongyu Zhang, Huifang Huang, Yulei Sui, Guandong Xu, Zhiyuan Shao, and Hai Jin. Cross-language binary-source code matching with intermediate representations. In SANER, 2022. find
  23. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… find
  24. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196, 2024. find
  25. Matthew R Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd M Austin, Trevor Mudge, and Richard B Brown. MiBench: A free, commercially representative embedded benchmark suite. In WWC. IEEE, 2001. find
  26. Iman Hosseini and Brendan Dolan-Gavitt. Beyond the C: Retargetable Decompilation using Neural Machine Translation. arXiv:2212.08950, 2022. — [arXiv:2212.08950](https://arxiv.org/abs/2212.08950) find
  27. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620, 2023. — [arXiv:2308.10620](https://arxiv.org/abs/2308.10620) find
  28. Tarindu Jayatilaka, Hideto Ueno, Giorgis Georgakoudis, EunJung Park, and Johannes Doerfert. Towards Compile-Time-Reducing Compiler Optimization Selection via Machine Learning. In ICPP, 2021. find
  29. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A Survey on Large Language Models for Code Generation. arXiv:2406.00515, 2024.
  30. Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. Unsupervised Translation of Programming Languages. arXiv:2006.03511, 2020.
  31. Chris Lattner and Vikram Adve. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In CGO, 2004. find
  32. Hugh Leather and Chris Cummins. Machine Learning in Compilers: Past, Present and Future. In FDL, 2020. find
  33. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — Competition-Level Code Generation with AlphaCode
  34. Youwei Liang, Kevin Stone, Ali Shameli, Chris Cummins, Mostafa Elhoushi, Jiadong Guo, Benoit Steiner, Xiaomeng Yang, Pengtao Xie, Hugh Leather, and Yuandong Tian. Learning Compiler Pass Orders using Coreset and Normalized Value Prediction. In ICML, 2023. find
  35. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv:1711.05101, 2017. find
  36. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. StarCoder 2 and The Stack v2: The Next Generation. arXiv:2402.19173, 2024. find
  37. Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. Learning performance-improving code edits. arXiv:2302.07867, 2023. find
  38. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In FAT, pp. 220–229. ACM, 2019. find
  39. William F. Ogilvie, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. Minimizing the Cost of Iterative Compilation with Active Learning. In CGO, 2017. find
  40. OpenAI. GPT-4 Technical Report. arXiv:2303.08774, 2023.
  41. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In ACL, 2002.
  42. Indraneil Paul, Goran Glavaš, and Iryna Gurevych. IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators. arXiv:2403.03894, 2024. find
  43. Phitchaya Mangpo Phothilimthana, Amit Sabne, Nikhil Sarda, Karthik Srinivasa Murthy, Yanqi Zhou, Christof Angermueller, Mike Burrows, Sudip Roy, Ketan Mandke, Rezsa Farahani, et al. A Flexible Approach to Autotuning Multi-pass Machine Learning Compilers. In PACT, 2021. find
  44. LLVM PM. Using the New Pass Manager — LLVM 17.0.6 documentation, 2021. URL https://llvm.org/docs/ NewPassManager.html. ↗
  45. Nilton Luiz Queiroz Jr and Anderson Faustino da Silva. A graph-based model for build optimization sequences: A study of optimization sequence length impacts on code size and speedup. COLA, 74, 2023. find
  46. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — Code Llama: Open Foundation Models for Code
  47. Eric Schulte, Jason Ruchti, Matt Noonan, David Ciarletta, and Alexey Loginov. Evolving Exact Decompilation. In BAR, 2018. find
  48. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. Adaptive Test Generation Using a Large Language Model. arXiv:2302.06527, 2023. — [arXiv:2302.06527](https://arxiv.org/abs/2302.06527) find
  49. Volker Seeker, Chris Cummins, Murray Cole, Björn Franke, Kim Hazelwood, and Hugh Leather. Revealing Compiler Heuristics Through Automated Discovery and Optimization. In CGO, 2024. find
  50. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 2024.
  51. Marc Szafraniec, Baptiste Roziere, Francois Charton, Hugh Leather, Patrick Labatut, and Gabriel Synnaeve. Code Translation with Compiler Representations. arXiv:2207.03578, 2022. — [arXiv:2207.03578](https://arxiv.org/abs/2207.03578) find
  52. Jubi Taneja, Avery Laird, Cong Yan, Madan Musuvathi, and Shuvendu K. Lahiri. LLM-Vectorizer: LLM-based Verified Loop Vectorizer. arXiv:2406.04693, 2024. — [arXiv:2406.04693](https://arxiv.org/abs/2406.04693) find
  53. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a.
  54. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023b.
  55. Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choromanski, and David Li. MLGO: a Machine Learning Guided Compiler Optimizations Framework. arXiv:2101.04808, 2021.
  56. Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023. find
  57. Zheng Wang and Michael O’Boyle. Machine Learning in Compiler Optimisation. arXiv:1805.03441, 2018. — [arXiv:1805.03441](https://arxiv.org/abs/1805.03441) find
  58. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering Code Generation with OSS-Instruct. arXiv:2312.02120, 2024. find
  59. Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Universal Fuzzing via Large Language Models. arXiv:2308.04748, 2023a. — [arXiv:2308.04748](https://arxiv.org/abs/2308.04748) find
  60. Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated Program Repair in the Era of Large Pre-Trained Language Models. In ICSE, 2023b. find
  61. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X. arXiv:2303.17568, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cummins-2024-meta-large-language-model-compiler/