Elamur Research⌕
☰
home / papers / 2026 / CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

Paper · 2026 · Sizhe Wang, Zhengren Wang, Dongsheng Ma et al.

Summary

Most code-generation benchmarks test LLMs in a single turn, but real software is built step by step from modular functions that reuse earlier ones. The authors call this incremental, reuse-driven process "codeflow" and present CodeFlowBench as the first benchmark for evaluating it.

An automated pipeline based on function dependency analysis breaks monolithic solutions into ordered subproblems, each with a verified solution and unit tests. The benchmark has two parts:

  • CodeFlowBench-Comp draws on Codeforces problems and is refreshed periodically to limit contamination.
  • CodeFlowBench-Repo draws on GitHub code collected through DomainEval.

The evaluation framework compares multi-turn and single-turn generation. It also adds structural metrics computed from dependency trees.

Across 19 LLMs, accuracy drops sharply in the multi-turn setting, by up to 50%. Scores also decline as dependency structures grow more complex. This shows that even strong reasoning models struggle to reuse earlier code and keep the overall goal in view, and it gives researchers an updatable testbed for workflows closer to real development.

Key points

  • CodeFlowBench-Comp contains over 5,000 Codeforces problems, maintained by an automated update pipeline. CodeFlowBench-Repo is built from real GitHub repositories via DomainEval.
  • New metrics: Average Pass Depth (APD@k) and Dependency Structure Complexity (DSC), both derived from function dependency trees.
  • Main result: among the 19 LLMs tested, multi-turn performance falls by as much as 50% compared with single-turn. Pass@1 decreases as DSC increases.
  • Error analysis: failures fall into three categories labeled IR, IG, and Instruction Misinterpretation (IM).
    • Comp errors are dominated by IR, while Repo shifts the balance toward IG and IM.
    • DeepSeek-V3 and o1-mini are limited mainly by IM; for o1-mini on Repo, IM accounts for 55.9% of errors.
    • For Gemini-3-Flash, IR is the main bottleneck.
  • Stated limitations: the data preparation step is specific to its sources, and no models fine-tuned for codeflow exist yet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksCode & programming languages · 129 works

Cite

Cite as Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu li, Feiyu Xiong and Wentao Zhang. 2026. CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.201

@inproceedings{wang2026codeflowbench,
  title = {{CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation}},
  author = {Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu li, Feiyu Xiong and Wentao Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.201},
  url = {https://doi.org/10.18653/v1/2026.acl-long.201}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pekka Abrahamsson, Outi Salo, Jussi Ronkainen, and Juhani Warsta. 2017. Agile software development methods: Review and analysis. arXiv preprint arXiv:1709.08439. — [arXiv:1709.08439](https://arxiv.org/abs/1709.08439) find
  2. Anthropic. 2025. Claude-4 system card. System card, Anthropic. find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Adam Boduch. 2019. React material-ui cookbook: build captivating user experiences using react and material-ui. Packt Publishing Ltd. find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  6. W Alec Cram. 2019. Agile development in practice: Lessons from the trenches. Information Systems Management, 36(1):2–14. find
  7. Daniel Feitosa, Apostolos Ampatzoglou, Antonios Gko-rtzis, Stamatia Bibi, and Alexander Chatzigeorgiou. 2020. Code reuse in practice: Benefiting or harming technical debt. Journal of Systems and Software, 167:110618. find
  8. Google. 2025. Gemini-3-flash system card. System card, Google. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065. llm-based agents for software engineering: A survey of current, challenges and future. arXiv…
  11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  12. Craig Larman. 2004. Agile and iterative development: a manager’s guide. Addison-Wesley Professional. find
  13. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  14. Stefan Haefliger, Georg Von Krogh, and Sebastian Spaeth. 2008. Code reuse in open source software. Management science, 54(1):180–193. find
  15. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938.
  16. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems, 37:11506– 11544. find
  17. Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010. find
  18. Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403. find
  19. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  20. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.
  21. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  22. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, and 1 others. 2024. Devbench: A comprehensive benchmark for software development. CoRR. find
  23. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  24. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  25. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024b. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977. — [arXiv:2409.02977](https://arxiv.org/abs/2409.02977) find
  26. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. find
  27. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. find
  28. OpenAI. 2025. Gpt-5 system card. System card, OpenAI. find
  29. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, and 1 others. 2025. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257.
  30. Zhengren Wang, Rui Ling, Chufan Wang, Yongan Yu, Zhiyu Li, Feiyu Xiong, and Wentao Zhang. 2025. Maintaincoder: Maintainable code generation under dynamic requirements. arXiv preprint arXiv:2503.24260. — [arXiv:2503.24260](https://arxiv.org/abs/2503.24260) find
  31. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120. find
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  33. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826–23854.
  34. Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2024. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199. — [arXiv:2412.21199](https://arxiv.org/abs/2412.21199) find
  35. Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Shing-Chi Cheung. 2025. Domaineval: An auto-constructed benchmark for multi-domain code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26148–26156. find
  36. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-codeflowbench-multi-turn-iterative-benchmark/