Elamur Research⌕
☰
home / papers / 2023 / AutoAgents: A Framework for Automatic Agent Generation

AutoAgents: A Framework for Automatic Agent Generation

Paper · 2023 · Guangyao Chen, Siwei Dong, Yu Shu et al.

Summary

Most LLM-based multi-agent systems use agents whose roles people define in advance. This makes them hard to adapt to new task types and expensive to scale. AutoAgents removes that dependence by assembling a task-specific team of expert agents automatically.

The framework runs in two phases. In the Drafting Stage, three fixed helper agents (Planner, Agent Observer and Plan Observer) discuss the input task. Together they decide which specialists to create and write an execution plan. In the Execution Stage, the generated specialists carry out the plan. Each one improves its own output through self-refinement, and they pool expertise through collaborative refinement. An Action Observer coordinates the team and uses a dynamic memory of earlier actions.

On two challenging tasks, including trivia creative writing, AutoAgents produced more coherent and accurate results than other agent-generation frameworks such as AgentVerse and SPP. A Tetris software case study shows practical use. The work suggests that fitting roles to each task and structuring team cooperation help LLMs with knowledge-heavy, cross-domain problems.

Key points

  • The pipeline has a Drafting Stage, with Planner, Agent Observer and Plan Observer, and an Execution Stage, coordinated by an Action Observer with dynamic long-term memory.
  • Unlike AutoGPT, BabyAGI, MetaGPT, Camel or AutoGen, it generates an unlimited number of agents dynamically. Compared with SPP and AgentVerse, it adds both self-refinement and collaborative refinement actions.
  • Ablations on trivia creative writing (Table 4) measured these drops:
    • about 3% without self-refinement
    • about 2% without collaborative refinement
    • about 1% without dynamic memory
  • In the case study, the system built a Python Tetris game by generating game design, UI design, programming and debugging experts.
  • The code is available at github.com/Link-AGI/AutoAgents (arXiv:2309.17288).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓Hong Kong University of Science and TechnologyHong Kong · China170 works · 26,341 citations🎓University of WaterlooWaterloo · Canada80 works · 15,030 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China46 works · 8,682 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Borje F. Karlsson, Jie Fu and Yemin Shi. 2023. AutoAgents: A Framework for Automatic Agent Generation. arXiv:2309.17288.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.17288 · arXiv 2309.17288

@misc{chen2023autoagents,
  title = {{AutoAgents: A Framework for Automatic Agent Generation}},
  author = {Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Borje F. Karlsson, Jie Fu and Yemin Shi},
  year = {2023},
  doi = {10.48550/arxiv.2309.17288},
  eprint = {2309.17288},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.17288}
}

References

41 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brigid Barron. Achieving coordination in collaborative problem-solving groups. The journal of the learning sciences, 9(4):403–436, 2000. 1 find
  2. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 1
  3. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023. 6
  4. Guangyao Chen, Peixi Peng, Yangru Huang, Mengyue Geng, and Yonghong Tian. Adaptive discovering and merging for incremental novel class discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11276–11284, 2024. 21 find
  5. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2023. 2, 4, 5, 6, 17
  6. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. 10 find
  7. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. 1, 3
  8. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142, 2023. 2
  9. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2023. 10
  10. Significant Gravitas. Auto-gpt: An autonomous gpt-4 experiment, 2023. URL https://github.com/Significant-Gravitas/Auto-GPT, 2023. 2, 5, 7 ↗
  11. Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. arXiv preprint arXiv:2304.12998, 2023. 1, 3 — [arXiv:2304.12998](https://arxiv.org/abs/2304.12998) find
  12. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. 2, 3, 5, 6
  13. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 10 find
  14. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611. Association for Computational Linguisti…
  15. Martin Josifoski, Lars Klein, Maxime Peyrard, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. Flows: Building blocks of reasoning and collaborating ai. arXiv preprint arXiv:2308.01285, 2023. 3 — [arXiv:2308.01285](https://arxiv.org/abs/2308.01285) find
  16. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023. 2, 3
  17. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023. 1, 3
  18. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. 10 find
  19. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online, July 2020. Association for Computational Linguistics. 1
  20. Y Nakajima. Task-driven autonomous agent utilizing gpt-4, pinecone, and langchain for diverse applications. See https://yoheinakajima.com/task-driven-autonomous-agent-utilizing-gpt-4-pinecone-and-langchain-for-diverse-applications (accessed 18 April 2023), 2023. 2, 3 ↗
  21. Laurie Miller Nelson. Collaborative problem solving. In Instructional-design theories and models, pages 241–267. Routledge, 2013. 1 find
  22. OpenAI. Gpt-4 technical report, 2023. 1
  23. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. 2 find
  24. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022. 2, 4,…
  25. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023. 6
  26. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023. 1
  27. Jeremy Roschelle and Stephanie D Teasley. The construction of shared knowledge in collaborative problem solving. In Computer supported collaborative learning, pages 69–97. Springer, 1995. 1 find
  28. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. 10
  29. Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. Llasm: Large language and speech model. arXiv preprint arXiv:2308.15930, 2023. 1 — [arXiv:2308.15930](https://arxiv.org/abs/2308.15930) find
  30. Steven A Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1):3, 1996. 1 find
  31. Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023. 3
  32. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698, 2023. 1
  33. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. 9 find
  34. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023. 1, 2, 3, 4, 5, 6, 9, 10, 17
  35. Ross Williams, Niyousha Hosseinichimeh, Aritra Majumdar, and Navid Ghaffarzadegan. Epidemic modeling with generative agents. arXiv preprint arXiv:2307.04986, 2023. 2, 4, 17 — [arXiv:2307.04986](https://arxiv.org/abs/2307.04986) find
  36. Anita Williams Woolley, Ishani Aggarwal, and Thomas W Malone. Collective intelligence and group performance. Current Directions in Psychological Science, 24(6):420–424, 2015. 5 find
  37. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. 2, 3, 5
  38. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305.14688, 2023. 2, 4, 5
  39. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. 9, 10 find
  40. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 7, 10
  41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. 9

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-autoagents-framework-automatic-agent-generation/