Elamur Research⌕
☰
home / papers / 2024 / Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering

Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering

Paper · 2024 · Tal Ridnik, Dedy Kredo and Itamar Friedman

Summary

The paper tackles competitive-programming code generation. Here, long and detail-heavy problem statements trip up LLMs, and standard prompt tricks such as chain-of-thought bring little benefit. The authors propose AlphaCodium, which shifts effort from crafting one prompt to designing a multi-stage, test-driven workflow ("flow engineering"). The flow has two phases:

  • Pre-processing: the model restates the problem as bullet points, explains why the public examples produce their outputs, drafts and ranks a few candidate approaches, and writes extra AI-generated tests.
  • Code iterations: the model repeatedly executes and repairs its code, first against the public tests and then against the AI-generated ones.

The method needs no training. It works with off-the-shelf models such as GPT-4, GPT-3.5 and DeepSeek, and is evaluated on the CodeContests validation and test splits. With GPT-4 on validation, pass@5 rises from 19% with a tuned direct prompt to 44%. AlphaCodium also beats CodeChain and AlphaCode while using far fewer model calls. This suggests that structured, execution-grounded pipelines can replace costly fine-tuning and massive sampling.

Key points

  • Benchmark: CodeContests, built from Codeforces-style problems. Only the validation (107 problems) and test (165 problems) splits are used, and each problem is checked against roughly 200 hidden private tests.
  • Design principle: writing extra tests is easier than writing a correct solution. Other techniques include YAML-structured outputs, modular code, soft decisions with double validation, postponing commitments, and "test anchors."
  • Main result: GPT-4 pass@5 on validation goes from 19% to 44%. The comparison table lists AlphaCodium at 29% (pass@5) versus AlphaCode at 28% (pass@10@100K).
  • Cost: each solution takes about 15–20 LLM calls, so a pass@5 submission needs about 100. That is roughly four orders of magnitude fewer calls than AlphaCode.
  • Resource: the full implementation and evaluation scripts are released at github.com/Codium-ai/AlphaCodium.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Tal Ridnik, Dedy Kredo and Itamar Friedman. 2024. Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv:2401.08500.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.08500 · arXiv 2401.08500

@misc{ridnik2024code,
  title = {{Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering}},
  author = {Tal Ridnik, Dedy Kredo and Itamar Friedman},
  year = {2024},
  doi = {10.48550/arxiv.2401.08500},
  eprint = {2401.08500},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.08500}
}

References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 1
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 1, 3
  3. DeepSeek. Deepseek coder: Let the code write itself, 2023. 2, 3, 7 find
  4. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495, 2023. 1
  5. Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020. 2, 3, 5 find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ridnik-2024-code-generation-alphacodium-prompt-engineering/