Elamur Research⌕
☰
home / papers / 2026 / ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

Paper · 2026 · Jie Yang, Honglin Guo, Li Ji et al.

Summary

Most coding benchmarks for LLM agents check isolated code edits in sandboxes that are already set up, and they validate those edits with unit tests. Real backend work involves more than that. ABC-Bench tests agents on the whole backend workflow: exploring a repository, changing code, configuring the environment, writing a Dockerfile, deploying the service, and passing end-to-end HTTP API tests against the running container. The authors built tasks semi-automatically with ABC-Pipeline, which mines open-source GitHub repositories and adds human validation. Out of roughly 2,000 repositories, it produced 224 tasks covering 8 languages and 19 frameworks. Agents run in an outer container. The system then builds and launches their output in a separate inner container and scores it only by external API checks. Even the strongest system tested solves just 63.2% of tasks at pass@1. Environment setup and deployment are the main obstacles. ABC-Bench therefore offers a deployment-oriented testbed that exposes how far agents remain from production-ready backend engineering.

Key points

  • The benchmark has 224 tasks across 8 languages and 19 frameworks, such as ASP.NET Core, Express, Spring Boot, Laravel, and Ruby on Rails.
    • 132 tasks focus on logic inside a prepared runtime.
    • 92 tasks also require the agent to configure the environment and start a containerized service.
  • The best pass@1 score is 63.2%, and many models score far lower.
  • Prompt hints matter. Removing Dockerfile examples and extra notes lowered every model's score, for example DeepSeek-V3.2 dropped from 50.1% to 42.0%.
  • Failure types shift with model size:
    • Qwen3-8B had 76 Path Missing errors, compared with 19 for GPT-5.
    • Larger models fail mostly on logic, with 30 Logic Errors for GPT-5 and 49 for Qwen3-Coder-480B-A35B.
  • The authors plan to open-source the benchmark, dataset, and evaluation pipeline. They note that task coverage across languages is uneven and that running the pipeline is costly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓Shanghai Innovation InstituteShanghai · China34 works · 325 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng, Tao Gui and Xipeng Qiu. 2026. ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1142

@inproceedings{yang2026abc,
  title = {{ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development}},
  author = {Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng, Tao Gui and Xipeng Qiu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1142},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1142}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. Swe-bench+: Enhanced coding benchmark for llms. CoRR, abs/2410.06992. find
  2. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732.
  3. Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. pages 2188–2200. find
  4. Yuxuan Cai, Lu Chen, Qiaoling Chen, Yuyang Ding, Liwen Fan, Wenjie Fu, Yufei Gao, Honglin Guo, Pinxue Guo, Zhenhua Han, and 1 others. 2025. Nex-n1: Agentic models trained via a unified ecosystem for large-scale environment construction. arXiv preprint arXiv:2512.04987. — [arXiv:2512.04987](https://arxiv.org/abs/2512.04987) find
  5. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhaoxiang Zhang, and Zhoujun Li. 2025. Mceval: Massively multilingual code evaluation. In The Thi… find
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  7. Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Liu, Jerry Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, and 35 others. 2024. Fullstack bench: Evaluating llms as full stack coders. CoRR, abs/2412.0… — [arXiv:2412.00535](https://arxiv.org/abs/2412.00535) find
  8. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. Swe-bench pro: Can… — [arXiv:2509.16941](https://arxiv.org/abs/2509.16941) find
  9. Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae agent: An llm-based agent for software engineering with test-time scaling. CoRR, abs/2507.23370. — [arXiv:2507.23370](https://arxiv.org/abs/2507.23370) find
  10. Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, and Jie Fu. 2024. Codeeditorbench: Evaluating code editing capability of large language models. CoRR, abs/2404.03543. — [arXiv:2404.03543](https://arxiv.org/abs/2404.03543) find
  11. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-coder technical report. CoRR, abs/2409.12186.
  12. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe…
  13. Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. 2024. Devbench: A comprehensive benchmark for software development. CoRR, abs/2403.08604. — [arXiv:2403.08604](https://arxiv.org/abs/2403.08604) find
  14. Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. 2025a. Prompting large language models to tackle the full software development lifecycle: A case study. In Proceedings…
  15. Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025b. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computationa… find
  16. Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu, Ronghao Chen, Yangfan He, Daxin Jiang, Binxing Jiao, Chen Hu, and Huacan Wang. 2025. Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. CoRR, abs/2508.02085. — [arXiv:2508.02085](https://arxiv.org/abs/2508.02085) find
  17. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  18. Harshith Padigela, Chintan Shah, and Dinkar Juyal. 2025. Ml-dev-bench: Comparative analysis of AI agents on ML development workflows. CoRR, abs/2502.00964. — [arXiv:2502.00964](https://arxiv.org/abs/2502.00964) find
  19. Musfiqur Rahman, SayedHassan Khatoonabadi, and Emad Shihab. 2025. Beyond synthetic benchmarks: Evaluating LLM performance on real-world class-level code generation. CoRR, abs/2510.26130. — [arXiv:2510.26130](https://arxiv.org/abs/2510.26130) find
  20. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053. find
  21. Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, and Dahua Lin. 2024. Code needs comments: Enhancing code llms with comment augmentation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thaila… find
  22. Gemini Team. 2025a. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261.
  23. GLM Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foun…
  24. The Terminal-Bench Team. 2025b. Terminal-bench: A benchmark for ai agents in terminal environments. find
  25. Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanovic, Jingxuan He, and Martin T. Vechev. 2025. Baxbench: Can llms generate correct and secure backends? In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-… find
  26. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 2 others. 2025. Openhands: An open platform f…
  27. Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. 2025. Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646. — [arXiv:2511.13646](https://arxiv.org/abs/2511.13646) find
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  29. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing…
  30. Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025a. Utboost: Rigorous evaluation of coding agents on swe-bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 -August 1, 2025, pages 3762–3774.… find
  31. Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiaoping Zhang. 2025b. Humaneval pro and MBPP pro: Evaluating large language models on self-invoking code generation task. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 13253– 13279.… find
  32. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving… — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  33. Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng, Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, and Qianxiang Wang. 2024. Swe-bench-java: A github issue resolving benchmark for j… — [arXiv:2408.14354](https://arxiv.org/abs/2408.14354) find
  34. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024a. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. pages 13643–13658.
  35. Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong, and Jie Tang. 2024b. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user queries. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virt…
  36. Yaolun Zhang, Yinxu Pan, Yudong Wang, Jie Cai, Zhi Zheng, Guoyang Zeng, and Zhiyuan Liu. 2024c. Pybench: Evaluating LLM agent on various real-world coding tasks. CoRR, abs/2407.16732. — [arXiv:2407.16732](https://arxiv.org/abs/2407.16732) find
  37. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Sys…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-abc-bench-benchmarking-agentic-backend/