Elamur Research⌕
☰
home / papers / 2024 / Language Agents as Optimizable Graphs

Language Agents as Optimizable Graphs

Paper · 2024 · Mingchen Zhuge, Wenyi Wang, Louis Kirsch et al.

Summary

Prompting schemes for LLM agents, such as chain of thought, tree of thought, Reflexion and multi-agent role play, each tend to arrive with their own code base and hand-crafted workflow. That makes them hard to combine or improve systematically. GPTSwarm treats any language agent as a directed acyclic computational graph. Nodes are operations like an LLM call, a web search or a tool invocation, and edges pass text between them. Several agent graphs can be merged into a composite "swarm" graph, where edges between agents serve as communication channels.

The authors propose two automatic optimizers:

  • Node optimization rewrites each node's prompt from its own input/output history, in the spirit of OPRO.
  • Edge optimization turns the discrete choice of inter-agent links into one probability per candidate edge and tunes these with REINFORCE, sampling only acyclic graphs.

Experiments on MMLU, Mini CrossWords, HumanEval and GAIA indicate that optimized graphs can beat hand-designed configurations. Exact scores are in a section missing from the supplied text. The work matters because it turns agent design from manual engineering into a learnable search problem.

Key points

  • Formalism: an agent is a tuple (N, E, F, o) executed in topological order, with the answer taken from output node o. COT, TOT and Self-Consistency all fit this representation.
  • Edge search: d candidate edges give 2^d configurations. The authors instead maximize expected utility under a distribution D_θ with θ ∈ [0,1]^d. Edges that would create a cycle are skipped during sampling, and REINFORCE is used because the utility is non-differentiable.
  • Comparison with DSPy: DSPy searches a Cartesian product of per-node candidates. GPTSwarm instead improves nodes one at a time, conditioned on the graph's execution history.
  • Experiments: benchmarks are MMLU, Mini CrossWords, HumanEval and GAIA, with web access through SearchApi. One demonstration shows Tree of Thought and Reflexion being recombined automatically via edge optimization.
  • Resource: an open-source framework (gptswarm.org) from KAUST and IDSIA, published at ICML 2024 (PMLR 235).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia30 works · 1,270 citations🎓Università della Svizzera italianaLugano · Switzerland10 works · 8,004 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland10 works · 8,825 citations🎓University of Applied Sciences and Arts of Southern SwitzerlandManno · Switzerland8 works · 7,374 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin and Jürgen Schmidhuber. 2024. Language Agents as Optimizable Graphs. arXiv:2402.16823.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.16823 · arXiv 2402.16823

@misc{zhuge2024language,
  title = {{Language Agents as Optimizable Graphs}},
  author = {Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin and Jürgen Schmidhuber},
  year = {2024},
  doi = {10.48550/arxiv.2402.16823},
  eprint = {2402.16823},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.16823}
}

References

60 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Auer, P., Cesa-Bianchi, N., and Fischer, P. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256, 2002. find
  3. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. find
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  5. Chase, H. LangChain. https://github.com/hwchase17/langchain, 2022. ↗
  6. Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., and Shi, Y. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288, 2023a.
  7. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  8. Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie, R., et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2023b.
  9. Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E. P., and Hu, Z. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548, 2022.
  10. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
  11. Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019. find
  12. Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.
  13. Gravitas, S. Auto-gpt. GitHub repository, 2023. find
  14. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
  15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
  16. Hong, S., Zheng, X., Chen, J., Cheng, Y., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
  17. Irie, K., Schlag, I., Csordás, R., and Schmidhuber, J. A modern self-referential weight matrix that learns to modify itself. In International Conference on Machine Learning, pp. 9660–9677. PMLR, 2022. find
  18. Ivakhnenko, A., Lapa, V., and ENGINEERING., P. U. L. I. S. O. E. Cybernetic Predicting Devices. JPRS 37, 803. Joint Publications Research Service [available from the Clearinghouse for Federal Scientific and Technical Information], 1965. URL https://books.google.com.sa/books?id=l38DHQAACAAJ. ↗
  19. Ivakhnenko, A. G. The group method of data handling, a rival of the method of stochastic approximation. Soviet Automatic Control, 13(3):43–55, 1968. find
  20. Kennedy, J. Swarm intelligence. In Handbook of nature-inspired and innovative computing: integrating classical models with emerging technologies, pp. 187–219. Springer, 2006. find
  21. Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023.
  22. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  23. Kirsch, L. and Schmidhuber, J. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34:14122–14134, 2021. find
  24. Kirsch, L. and Schmidhuber, J. Eliminating meta optimization through self-referential meta learning. arXiv preprint arXiv:2212.14392 and First Conference on Automated Machine Learning (Workshop), 2022. — [arXiv:2212.14392](https://arxiv.org/abs/2212.14392) find
  25. Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022. — [arXiv:2212.04458](https://arxiv.org/abs/2212.04458) find
  26. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199–22213, 2022. langchain ai. LangGraph. https://github.com/langchain-ai/langgraph, 2024. ↗
  27. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023.
  28. Liu, J. LlamaIndex, 11 2022. URL https://github.com/jerryjliu/llama_index. ↗
  29. Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023.
  30. Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023.
  31. Minsky, M. Society of mind. Simon and Schuster, 1988. find
  32. Nakajima, Y. Babyagi. Python. https://github.com/yoheinakajima/babyagi, 2023. ↗
  33. Nepusz, T. and Vicsek, T. Hierarchical self-organization of non-cooperating individuals. Plos one, 8(12):e81449, 2013. find
  34. Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1–22, 2023. find
  35. Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023.
  36. Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., and Sun, M. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023.
  37. Schmidhuber, J. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987. find
  38. Schmidhuber, J. A ‘self-referential’weight matrix. In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September 1993 3, pp. 446–450. find
  39. Schmidhuber, J. Deep learning in neural networks: An overview. Neural networks, 61:85–117, 2015. find
  40. Sel, B., Al-Tawaha, A., Khattar, V., Wang, L., Jia, R., and Jin, M. Algorithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2308.10379, 2023. find
  41. Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 14, 2023.
  42. Torantulino et al. Auto-gpt. https://github.com/Significant-Gravitas/Auto-GPT, 2023. ↗
  43. Vowels, M. J., Camgoz, N. C., and Bowden, R. D’ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys, 55(4):1–36, 2022. find
  44. Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023.
  45. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  46. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  47. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. find
  48. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.
  49. XAgent Team. Xagent: An autonomous agent for complex task solving, 2023. find
  50. Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023.
  51. Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J., Zhao, J., Liu, Q., Liu, C., et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310.10634, 2023.
  52. Yan, L. C., Yoshua, B., and Geoffrey, H. Deep learning. nature, 521(7553):436–444, 2015. find
  53. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  54. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  55. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=5Xc1ecxO1h. ↗
  56. Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X.-J., and Qiu, X. Exchange-of-thought: Enhancing large language model capabilities through cross-model communication. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15135–15153, 2023. find
  57. Zelikman, E., Lorch, E., Mackey, L., and Kalai, A. T. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023. — [arXiv:2310.02304](https://arxiv.org/abs/2310.02304) find
  58. Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
  59. Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17066, 2023. — [arXiv:2305.17066](https://arxiv.org/abs/2305.17066) find
  60. A. The GPTSwarm Framework A.1. The Vision find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhuge-2024-language-agents-optimizable-graphs/