Elamur Research⌕
☰
home / papers / 2024 / EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning

EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning

Paper · 2024 · Dong Huang, Guangtao Zeng, Jianbo Dai et al.

Summary

Code-generating LLMs are usually tuned for correctness, but the programs they produce often run slower and use more memory than human-written reference solutions. This wastes compute and energy and is a problem on constrained hardware. The authors present EffiCoder, a fine-tuning approach that targets both correctness and efficiency. They build EffiInstruct by pooling tasks from open-source code datasets on Hugging Face, cleaning them, and adding test cases. Several LLMs then generate candidate solutions in multiple programming languages. Each candidate is run locally and profiled for runtime and memory, and the cheapest working version becomes the training target. A preliminary experiment showed that the efficiency of the training code closely tracks the efficiency of the code a model later generates. Fine-tuning on EffiInstruct raised pass@1 and roughly halved execution time on several models. It also beat the efficiency-focused baselines PIE and Mercury. The dataset, code and weights are released openly, offering a scalable way to make LLM-written code cheaper to run (ICML 2025).

Key points

  • Motivation: On EffiBench, GPT-4-Turbo's code ran 1.69× slower than canonical solutions on average and 45.49× slower in the worst case.
  • Preliminary study: Training-data efficiency correlated strongly with output efficiency, measured three ways:
    • ET (Execution Time): 0.972
    • MU (Max Memory Usage): 0.950
    • TMU (Total Memory Usage): 0.986
  • Qwen2.5-Coder-7B-Instruct:
    • pass@1 on EffiBench rose from 44.8% to 57.7%.
    • pass@1 on HumanEvalPlus rose from 76.2% to 78.0%.
    • Mean runtime on EffiBench fell from 0.31 s to 0.16 s (−48.4%).
  • CodeLlama-7B on HumanEvalPlus:
    • pass@1 reached 31.1% with EffiCoder, versus 19.5% for PIE and 9.1% for Mercury.
    • Runtime fell by 46.2% with EffiCoder, versus 23.1% with PIE.
  • Training data scale: EffiInstruct has more than 70,000 examples, compared with roughly 1,800 in Mercury's training set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksCode & programming languages · 129 works

Cite

Cite as Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo and Jie M. Zhang. 2024. EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning. arXiv:2410.10209.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.10209 · arXiv 2410.10209

@misc{huang2024efficoder,
  title = {{EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning}},
  author = {Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo and Jie M. Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2410.10209},
  eprint = {2410.10209},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.10209}
}

References

73 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sustainability: By reducing computational resource consumption (e.g., energy and memory usage), our work aligns with global efforts to mitigate the environmental impact of large-scale computing. Efficient code generation could lower operational costs and energy demands in industries reliant on softw… find
  2. Resource-Constrained Environments: Enhanced efficiency enables broader adoption of LLM-generated code in mobile, embedded, or edge computing systems, where energy and processing power are limited. Ahmed, T. and Devanbu, P. T. Few-shot training llms for project-specific code-summarization. In 37th IE… — Evaluating Large Language Models Trained on Code
  3. International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pp. 177:1–177:5. find
  4. ACM, 2022. doi: 10. 1145/3551349.3559555. URL https://doi.org/ 10.1145/3551349.3559555. — [doi:10.1145/3551349.3559555](https://doi.org/10.1145/3551349.3559555) find
  5. Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., Umapathi, L. K., Anderson, C. J., Zi, Y., Lamy-Poirier, J., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, F. D., del Río, B. G., Liu, Q., Bose, S., Bh… — SantaCoder: don't reach for the stars!
  6. Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. CoRR, abs/2304.05128, 2023. doi: 10.48550/ARXIV.2304.05128. URL https://doi.org/10.48550/arXiv.2304.05128. ↗
  7. Computations, C. Dolphin coder. https://huggingface.co/datasets/ cognitivecomputations/dolphin-coder, 2023. ↗
  8. Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic.com/ news/claude-3-family. ↗
  9. Computer, T. Glaive-code-assistant. https://huggingface.co/datasets/Together/ Glaive-Code-Assistant, 2023. ↗
  10. Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732. ↗
  11. Dai, J., Lu, J., Feng, Y., Ruan, R., Cheng, M., Tan, H., and Guo, Z. MHPP: exploring the capabilities and limitations of language models beyond basic code generation. CoRR, abs/2405.11430, 2024. doi: 10.48550/ARXIV. 2405.11430. URL https://doi.org/10.48550/ arXiv.2405.11430. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  12. BigCode. Self-oss-instruct-sc2-exec-filter-50k. https://huggingface.co/datasets/bigcode/ self-oss-instruct-sc2-exec-filter-50k, 2023. ↗
  13. DeepSeekAI. Deepseek coder: Let the code write itself, 2023. URL https://deepseekcoder.github.io/. ↗
  14. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — Language Models are Few-Shot Learners
  15. Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., and Zhang, L. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. CoRR, abs/2304.02014, 2023. doi: 10.48550/ARXIV. 2304.02014. URL https://doi.org/10.48550/ arXiv.2304.02014. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  16. Du, M., Luu, A. T., Ji, B., and Ng, S. Mercury: An efficiency benchmark for LLM code synthesis. CoRR, abs/2402.07844, 2024. doi: 10.48550/ARXIV. 2402.07844. URL https://doi.org/10.48550/ arXiv.2402.07844. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  17. Chaudhary, S. Code alpaca: An instruction-following llama model for code generation. https://github.com/ sahil280114/codealpaca, 2023. ↗
  18. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  19. OpenReview.net, 2023. URL https://openreview.net/pdf?id=hQwb-lbM6EL. ↗
  20. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… find
  21. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319–18345. PMLR, 2023. find
  22. Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 919–931. find
  23. IEEE, 2023. doi: 10.1109/ ICSE48619.2023.00085. URL https://doi.org/ 10.1109/ICSE48619.2023.00085. — [doi:10.1109/icse48619.2023.00085](https://doi.org/10.1109/icse48619.2023.00085) find
  24. Haque, M. M. A., Ahmad, W. U., Lourentzou, I., and Brown, C. Fixeval: Execution-based evaluation of program fixes for competitive programming problems. CoRR, abs/2206.07796, 2022. doi: 10.48550/ARXIV. 2206.07796. URL https://doi.org/10.48550/ arXiv.2206.07796. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  25. Hasan, M., Muttaqueen, T., Ishtiaq, A. A., Mehrab, K. S., Haque, M. M. A., Hasan, T., Ahmad, W. U., Iqbal, A., and Shahriyar, R. Codesc: A large code-description parallel dataset. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL/IJC… find
  26. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring coding challenge competence with apps. NeurIPS, 2021.
  27. Huang, D., Dai, J., Weng, H., Wu, P., Qing, Y., Zhang, J. M., Cui, H., and Guo, Z. SOAP: enhancing efficiency of generated code via self-optimization. CoRR, abs/2405.15189, 2024a. doi: 10.48550/ARXIV.2405.15189. URL https://doi.org/10.48550/arXiv.2405.15189. — [doi:10.48550/arxiv.2405.15189](https://doi.org/10.48550/arxiv.2405.15189) ↗
  28. Huang, D., Qing, Y., Shang, W., Cui, H., and Zhang, J. M. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037, 2024b. — [arXiv:2402.02037](https://arxiv.org/abs/2402.02037) find
  29. Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., Yang, A., Men, R., Huang, F., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-coder technical report. 2024. URL https://api.semanticscholar.org/CorpusID:272707390. ↗
  30. Jiang, N., Liu, K., Lutellier, T., and Tan, L. Impact of code language models on automated program repair. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 1430–1442. find
  31. IEEE, 2023. doi: 10.1109/ ICSE48619.2023.00125. URL https://doi.org/ 10.1109/ICSE48619.2023.00125. — [doi:10.1109/icse48619.2023.00125](https://doi.org/10.1109/icse48619.2023.00125) find
  32. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M., Umapathi, L. K.,… find
  33. Li, Y., Choi, D. H., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli,… — Competition-Level Code Generation with AlphaCode
  34. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309.05463, 2023b. doi: 10.48550/ARXIV.2309.05463. URL https://doi.org/10.48550/arXiv.2309.05463. ↗
  35. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.
  36. Liu, J., Xie, S., Wang, J., Wei, Y., Ding, Y., and Zhang, L. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450, 2024b. — [arXiv:2408.06450](https://arxiv.org/abs/2408.06450) find
  37. Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurI…
  38. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  39. OpenReview.net, 2024. URL https://openreview.net/forum?id=UnUwSIgK5W. ↗
  40. Rozière, B., Lachaux, M., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 202…
  41. MAP. Codefeedback-filtered-instruction. https://huggingface.co/datasets/m-a-p/ CodeFeedback-Filtered-Instruction, 2023. ↗
  42. Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai.meta.com/blog/meta-llama-3/. ↗
  43. Mir, A. M., Latoskinas, E., Proksch, S., and Gousios, G. Type4py: Practical deep similarity learning-based type inference for python. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 2241–2252. find
  44. ACM, 2022. doi: 10.1145/3510003.3510124. URL https://doi.org/10.1145/3510003.3510124. — [doi:10.1145/3510003.3510124](https://doi.org/10.1145/3510003.3510124) ↗
  45. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Canton-Ferrer, C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T.,… — Code Llama: Open Foundation Models for Code
  46. Shi, J., Yang, Z., and Lo, D. Efficient and green large language models for software engineering: Vision and the road ahead. arXiv preprint arXiv:2404.04566, 2024. — [arXiv:2404.04566](https://arxiv.org/abs/2404.04566) find
  47. Muennighoff, N., Liu, Q., Zebaze, A. R., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  48. OpenReview.net, 2024. URL https://openreview.net/forum?id=mw1PWNSWZP. ↗
  49. Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., and Yazdanbakhsh, A. Learning Performance-Improving Code Edits. In The Twelfth International Conference on Learning Representations (ICLR), 2024. find
  50. Sun, Q., Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734, 2024. — [arXiv:2403.14734](https://arxiv.org/abs/2403.14734) find
  51. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  52. OpenReview.net, 2023. URL https://openreview.net/pdf?id=iaYcJKpY2B_. ↗
  53. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  54. OpenAI. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗
  55. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/arXiv… ↗
  56. Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. find
  57. Open-Review.net, 2024b. URL https://openreview.net/forum?id=XUeoOBid3x. ↗
  58. UIUC, I. Magicoder-evol-instruct-110k. https://huggingface.co/datasets/ise-uiuc/ Magicoder-Evol-Instruct-110K, 2023a. ↗
  59. UIUC, I. Magicoder-oss-instruct-75k. https://huggingface.co/datasets/ise-uiuc/ Magicoder-OSS-Instruct-75K, 2023b. ↗
  60. Xia, C. S., Deng, Y., and Zhang, L. Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint arXiv:2403.19114, 2024. — [arXiv:2403.19114](https://arxiv.org/abs/2403.19114) find
  61. Vezora. Tested-143k-python-alpaca. https://huggingface.co/datasets/Vezora/ Tested-143k-Python-Alpaca, 2023. ↗
  62. Wang, Y., Wang, W., Joty, S. R., and Hoi, S. C. H. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Proces… find
  63. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Li… find
  64. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
  65. OpenReview.net, 2024. URL https://openreview.net/forum?id=CfXh93NDgH. ↗
  66. Zhao, C., Jia, X., Viswanathan, V., Wu, T., and Neubig, G. Self-guide: Better task-specific instruction following via self-synthetic finetuning. arXiv preprint arXiv:2407.12874, 2024. find
  67. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673–56…
  68. Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Associati…
  69. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
  70. OpenReview.net, 2022. URL https://openreview.net/forum? id=gEZrGCozdqR. ↗
  71. Wei, J., Durrett, G., and Dillig, I. Typet5: Seq2seq type inference using static analysis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  72. OpenReview.net, 2023. URL https://openreview.net/ pdf?id=4TyNEhI2GdN. ↗
  73. Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., Von Werra, L., Guha, A., and Zhang, L. Selfcodealign: Self-alignment for code generation. arXiv preprint arXiv:2410.24198, 2024a.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2024-efficoder-enhancing-code-generation-large/