Elamur Research⌕
☰
home / papers / 2024 / Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents

Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents

Paper · 2024 · Yifan Song, Da Yin, Xiang Yue et al.

Summary

Open-source LLMs used as agents still trail GPT-4. The usual remedy is behavioral cloning (BC), meaning supervised fine-tuning (SFT) on expert trajectories. Because the model never sees the consequences of its own mistakes, the resulting policy can be weak and may generalize poorly.

The authors propose ETO (Exploration-based Trajectory Optimization). They first train an SFT base agent on ReAct-style expert demonstrations that include CoT rationales. This agent then attempts the training tasks in the environment. Any attempt that scores below the expert trajectory is paired with it, and the policy is updated on these contrastive pairs using the DPO loss. The cycle can repeat, with each newly tuned agent producing fresh failures. This offline, iterative design avoids the instability and cost of online RL methods such as PPO.

On WebShop, ScienceWorld and ALFWorld, ETO clearly beats SFT and other strong baselines. It also reaches higher rewards in fewer action steps and remains useful when expert data is missing. The work shows that failed attempts provide valuable training signal for building capable open agents.

Key points

  • Benchmarks: WebShop (web navigation), ScienceWorld (simulated science experiments) and ALFWorld (embodied household tasks). Agent tasks are framed as a POMDP whose final reward lies in [0, 1].
  • Generalization: On the out-of-distribution ScienceWorld test split, ETO improves on SFT by 22%.
  • Self-play with a BC base (Table 5): average reward was 63.1 for SFT, 63.6 for RFT and 67.4 for ETO.
  • Self-play without expert data, starting from Llama-2-7B-Chat:
    • Prompting the untuned model alone gives 17.9.
    • RFT reaches 48.4.
    • ETO by itself drops to 12.5.
    • RFT followed by ETO reaches 51.2.
  • Limitations and resources: Failure–success pairs ignore the step where an error actually begins, so there is no step-level reward. The agents are also single-task specialists. Code and data are released at github.com/Yifan-Song793/ETO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓Jiangsu Normal UniversityTongshan · China2 works · 90 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksEngineering · 133 worksControl and Systems Engineering · 45 works

Cite

Cite as Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li and Bill Yuchen Lin. 2024. Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.409

@inproceedings{song2024trial,
  title = {{Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents}},
  author = {Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li and Bill Yuchen Lin},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.409},
  url = {https://doi.org/10.18653/v1/2024.acl-long.409}
}

References

45 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. 2023. Rest meets react: Self-improvement for multi-step reasoning llm agent. ArXiv preprint, abs/2312.10003. — [arXiv:2312.10003](https://arxiv.org/abs/2312.10003) find
  2. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  3. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. ArXiv preprint, abs/2310.05915.
  4. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299–4307. find
  5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems.
  6. Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. 2019. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362– 369. find
  7. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. ArXiv preprint, abs/2308.08998.
  8. Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35. find
  9. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. ArXiv preprint, abs/2310.06825. find
  10. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. ArXiv preprint, abs/2305.20050.
  11. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. 2023. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems.
  12. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations.
  13. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  14. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. ArXiv, abs/2304.09842.
  15. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. ArXiv preprint, abs/2311.12983.
  16. Yohei Nakajima. 2023. Babyagi. find
  17. OpenAI. 2022. Introducing chatgpt. find
  18. OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.
  19. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  20. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. ArXiv preprint, abs/1910.00177. find
  21. Dean A Pomerleau. 1991. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88–97. find
  22. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. ArXiv preprint, abs/2305.18290.
  23. Toran Bruce Richards. 2023. Auto-gpt: An autonomous gpt-4 experiment. find
  24. Stephane Ross and J Andrew Bagnell. 2014. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979. find
  25. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627– 635. JMLR Workshop and Conference Proceedi… find
  26. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347. find
  27. Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model alignment: A survey. ArXiv preprint, abs/2309.15025.
  28. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10737–10746.…
  29. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations.
  30. Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world applications via restful apis. ArXiv preprint, abs/2306.06624.
  31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288.
  32. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023a. A survey on large language model based autonomous agents. ArXiv preprint, abs/2308.11432.
  33. Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. 2023b. Making large language models better reasoners with alignment. ArXiv preprint, abs/2309.02144. — [arXiv:2309.02144](https://arxiv.org/abs/2309.02144) find
  34. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023c. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. ArXiv preprint, abs/2312.08935.
  35. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. ScienceWorld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298, Abu Dhabi, United Arab Emirates. Association for Comp… find
  36. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  37. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  38. Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2023. Lumos: Learning agents with unified data, modular design, and open-source llms. ArXiv preprint, abs/2311.05657. find
  39. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. ArXiv preprint, abs/2308.01825.
  40. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. ArXiv preprint, abs/2310.12823.
  41. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. ArXiv preprint, abs/2307.13854.
  42. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023d. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In The Eleventh International Conference on Learning Representations.
  43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  44. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. ArXiv preprint, abs/2309.07864.
  45. Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lu-song Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. 2023. Embodied multi-modal agent trained by find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2024-trial-error-exploration-based-trajectory/