Elamur Research⌕
☰
home / papers / 2023 / Experiential Co-Learning of Software-Developing Agents

Experiential Co-Learning of Software-Developing Agents

Paper · 2023 · Chen Qian, Yufan Dang, Jiahao Li et al.

Summary

Multi-agent LLM systems such as ChatDev build software through dialogue between agents, but they handle every task in isolation, so the same mistakes and wasteful trial-and-error keep coming back. The authors propose Experiential Co-Learning, in which an instructor agent and an assistant agent learn together from their own earlier runs. First, the pair rehearses training tasks, and each run is logged as a chain of code versions linked by instructions (co-tracking). These logs are then turned into task execution graphs. Each node is scored with external signals: whether the code compiles, how close it is to the task, and how close it is to the final solution. Links that skip intermediate steps and lead to better code are saved as "shortcut" experiences in a separate pool for each role (co-memorizing). On new tasks, both agents retrieve matching shortcuts to improve their instructions and their code (co-reasoning). Tests on unseen software tasks show better quality, fewer repeated errors and less need for human intervention. The work is among the first to reuse experience across tasks in cooperating agents. Code and data are released.

Key points

  • The framework has three modules (co-tracking, co-memorizing, co-reasoning) and two roles (instructor, assistant). It is implemented in the OpenBMB/ChatDev repository on GitHub.
  • In the ablation, the full model holds 537 shortcuts per role and reaches a Quality of 0.7304. Removing the assistant's experiences lowers Quality to 0.5305, removing the instructor's lowers it to 0.6840, and removing both lowers it to 0.4267.
  • An ExpeL-like variant stores experiences from adjacent steps instead of shortcuts. It collects 1604 experiences per role but scores only 0.6398, which suggests that fewer, filtered shortcuts work better than many raw ones.
  • Two other variants also fall short of the full model: keeping only the single longest shortcut (0.6752) and skipping graph construction (0.6821).
  • Retrieval works best with k_code = 1 and k_text = 2, whereas the default is 1/1. The similarity threshold θ has little effect, because retrieved items usually exceed 0.85 similarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China439 works · 94,783 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China80 works · 33,443 citations🎓Dalian University of TechnologyDalian · China21 works · 70 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, Zhiyuan Liu and Maosong Sun. 2023. Experiential Co-Learning of Software-Developing Agents. arXiv:2312.17025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.17025 · arXiv 2312.17025

@misc{qian2023experiential,
  title = {{Experiential Co-Learning of Software-Developing Agents}},
  author = {Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, Zhiyuan Liu and Maosong Sun},
  year = {2023},
  doi = {10.48550/arxiv.2312.17025},
  eprint = {2312.17025},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.17025}
}

References

64 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Henri Barki, Suzanne Rivard, and Jean Talbot. 1993. Toward an Assessment of Software Development Risk. In Journal of Management Information Systems, volume 10, pages 203–225. find
  2. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large Language Models in Machine Translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858–867.
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. In arXiv preprint arXiv:2303.12712.
  5. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. Large Language Models as Tool Makers. In The Twelfth International Conference on Learning Representations (ICLR).
  6. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards Better LLM-based Evaluators through Multi-agent Debate. In The Twelfth International Conference on Learning Representations (ICLR).
  7. Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. 2023. GameGPT: Multi-agent Collaborative Framework for Game Development. In arXiv preprint arXiv:2310.08067.
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. In arXiv preprint arXiv:2107.03374.
  9. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. 2024. Agentverse: Facilitating Multi-agent Collaboration and Exploring Emergent Behaviors in Agents. In The Twelfth International Conference on Learning Representations (IC…
  10. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. LM vs LM: Detecting Factual Errors via Cross Examination. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12621–12640.
  11. Katherine Compton and Scott Hauck. 2002. Reconfigurable Computing: a Survey of Systems and Software. In ACM Computing Surveys (csuR), volume 34, pages 171–210. find
  12. Shiying Ding, Xinyi Chen, Yan Fang, Wenrui Liu, Yiwu Qiu, and Chunlei Chai. 2023. DesignGPT: Multi-Agent Collaboration in Design. In 2023 16th International Symposium on Computational Intelligence and Design (ISCID), pages 204–208. find
  13. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut Learning in Deep Neural Networks. In Nature Machine Intelligence, volume 2, pages 665–673. find
  14. Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. 2023. MindAgent: Emergent Gaming Interaction. In arXiv preprint arXiv:2309.09971. find
  15. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.
  16. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The T…
  17. Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2023. War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars. In arXiv preprint arXiv:2311.17227.
  18. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023. Benchmarking Large Language Models as AI Research Agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  19. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. In arXiv preprint arXiv:2001.08361.
  20. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. find
  21. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474.
  22. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a. CAMEL: Communicative Agents for ”Mind” Exploration of Large Language Model Society. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS).
  23. Yuan Li, Yixuan Zhang, and Lichao Sun. 2023b. Metaagents: Simulating Interactions of Human Behaviors for LLM-based Task-oriented Coordination via Collaborative Generative Agents. In arXiv preprint arXiv:2310.06500.
  24. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. 2023. Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis. In arXiv preprint arXiv:231…
  25. Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. 2023. find
  26. BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  27. Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. 2023. LASER: LLM agent with state-space exploration for web navigation. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  28. Harlan D Mills. 1976. Software development. In IEEE Transactions on Software Engineering, 4, pages 265– 273. find
  29. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11048–1…
  30. Anton Osika. 2023. GPT-Engineer. https://github.com/AntonOsika/gpt-engineer. ↗
  31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  32. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pages 1–22. find
  33. Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62st Annual Meeting of the Association for Computational Linguistics (ACL).
  34. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2024. Toolllm: Facilitating Large Language Models to Master 16000+ Real-World APIs. In The Twelfth International Conference on Learning Representations (ICLR).
  35. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2023. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. In Proceedings of the 2024 Conference of the North American Ch…
  36. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI blog, volume 1, page 9. find
  37. Toran Bruce Richards. 2023. AutoGPT. In https://github.com/Significant-Gravitas/AutoGPT. ↗
  38. Jingqing Ruan, YiHong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, du qing, shi shiwei, Hangyu Mao, Xingyu Zeng, and Rui Zhao. 2023. TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  39. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning To Retrieve Prompts for In-Context Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 2655–2671.
  40. In Yu Sasaki and Kazumaro Aoki. 2009. Finding Preim-ages in Full MD5 Faster Than Exhaustive Search. In Advances in Cryptology - EUROCRYPT 2009, pages 134–152. find
  41. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS).
  42. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role Play with Large Language Models. In Nature, volume 623, pages 493–498.
  43. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS).
  44. Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. 2024. Cognitive architectures for language agents. In Transactions on Machine Learning Research (TMLR). Survey Certification. find
  45. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971.
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30.
  47. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An Open-Ended Embodied Agent with Large Language Models. In Intrinsically-Motivated and Open-Ended Learning Workshop @NeurIPS2023. find
  48. Lei Wang, Jingsen Zhang, Hao Yang, Zhiyuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. 2023b. When Large Language Model based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm. In arXiv preprint…
  49. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023c. Large Language Models are not Fair Evaluators. In arXiv preprint arXiv:2305.17926. find
  50. Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023d. Avalon’s Game of Thoughts: Battle Against Deception through Recursive Contemplation. In Findings of the Association for Computational Linguistics: ACL 2024. find
  51. Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. 2024. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. In The Twelfth International Conference on Learning Representations (ICLR).
  52. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023e. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 13… find
  53. Zhilin Wang, Yu Ying Chiu, and Yu Cheung Chiu. 2023f. Humanoid Agents: Platform for Simulating Human-like Generative Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pages 167–176. find
  54. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent Abilities of Large Language Models. In Transactions o…
  55. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837.
  56. Lilian Weng. 2023. LLM-powered Autonomous Agents. In lilianweng.github.io. find
  57. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. In arXiv preprint arXi…
  58. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. In The Twelfth International Conference on Learning Representations (ICLR).
  59. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS).
  60. An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. 2023. On Generative Agents in Recommendation. In arXiv preprint arXiv:2310.10108. — [arXiv:2310.10108](https://arxiv.org/abs/2310.10108) find
  61. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM Agents Are Experiential Learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 17, pages 19632–19642. find
  62. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2024. Webarena: A realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations (ICLR).
  63. Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. 2023. Agents: An Open-source Framework for Autonomous Language Agents. I… find
  64. Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. 2023. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Mem…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2023-experiential-co-learning-software/