Elamur Research⌕
☰
home / papers / 2026 / CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

Paper · 2026 · Shidong Yang, Ziyu Ma, Tongwen Huang et al.

Summary

Reinforcement learning for LLM agents usually relies on a fixed training set, either expert demonstrations or offline LLM-generated tasks. Such a set does not change as the agent improves, so it misses many complex interaction cases. CoEvolve (AMAP, Alibaba Group) addresses this with a closed loop in which the agent and its training data evolve together.

During GRPO training, the system examines rollout trajectories for three weakness indicators:

  • forgetting: a task that was solved earlier now fails;
  • boundary: the agent sits at a decision boundary;
  • rare: long-tail cases.

These signals prompt an LLM to revisit the relevant parts of the environment. The interactions it finds are turned into task drafts, checked by executing them in the real environment, and then added to the training set. No human supervision is needed.

On AppWorld and BFCL, the method gives large absolute gains over the base models for Qwen2.5-7B, Qwen3-4B and Qwen3-30B-A3B, at about 10% extra compute. This makes adaptive, self-generated curricula a practical and inexpensive option for training tool-using agents.

Key points

  • Average absolute gains over the base models: +19.43% for Qwen2.5-7B, +15.58% for Qwen3-4B and +18.14% for Qwen3-30B-A3B, across AppWorld and BFCL. Code is available at github.com/AMAP-ML/CoEvolve.
  • Pipeline stages:
    • Stage 1: GRPO training plus signal extraction.
    • Stage 2: LLM re-exploration conditioned on the signals.
    • Stage 3: task abstraction and environment validation.
  • Signal analysis (Qwen3-4B):
    • Boundary signals make up the largest share of signals, 51.4% on AppWorld and 45.5% on BFCL.
    • Forgetting signals matter most in the ablation; removing them lowers the score from 49.36 to 45.18.
    • Removing boundary or rare signals costs about 1.6–1.9 points each.
  • Cost versus a GRPO baseline without the loop: feedback adds 9.67% (AppWorld) and 12.76% (BFCL) of training time. In return, AppWorld rises from 28.57 to 35.12 and BFCL from 58.00 to 63.00.
  • Cross-domain transfer (Qwen3-4B):
    • Training on AppWorld raises zero-shot BFCL from 26.50 to 45.00.
    • Training on BFCL raises zero-shot AppWorld from 16.67 to 19.04.
    • Synthesized tasks mostly have only moderate similarity (0.4–0.7) to validation tasks, so they are not near-duplicates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,172 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang and Xiangxiang Chu. 2026. CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1055

@inproceedings{yang2026coevolve,
  title = {{CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution}},
  author = {Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang and Xiangxiang Chu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1055},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1055}
}

References

58 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  3. Anthropic. 2025. Claude sonnet 4.5. https://docs.anthropic.com/claude/docs/models-overview. Accessed: 2025-01. ↗
  4. Peter Chen, Xi Chen, Wotao Yin, and Tianyi Lin. 2025a. Compo: Preference alignment via comparison oracles. arXiv preprint arXiv:2505.05465. — [arXiv:2505.05465](https://arxiv.org/abs/2505.05465) find
  5. Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen, and Tianyi Lin. 2025b. Stepwise guided policy optimization: Coloring your incorrect reasoning in grpo. arXiv preprint arXiv:2505.11595. — [arXiv:2505.11595](https://arxiv.org/abs/2505.11595) find
  6. Yihan Chen, Benfeng Xu, Xiaorui Wang, Yongdong Zhang, and Zhendong Mao. 2025c. Training llm-based agents with synthetic self-reflected trajectories and partial masking. arXiv preprint arXiv:2505.20023. — [arXiv:2505.20023](https://arxiv.org/abs/2505.20023) find
  7. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. 2025. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. — [arXiv:2504.02546](https://arxiv.org/abs/2504.02546) find
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  9. Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Luu Anh Tuan, and Shafiq Joty. 2024. Data augmentation using llms: Data perspectives, learning paradigms and challenges. In Findings of the Association for Computational Linguistics: ACL 2024, pages… find
  10. Hanxing Ding, Shuchang Tao, Liang Pang, Zihao Wei, Jinyang Gao, Bolin Ding, Huawei Shen, and Xueqi Cheng. 2025. Toolcoder: A systematic code-empowered tool learning framework for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume… find
  11. Chenhui Gou, Ziyu Ma, Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, and Hamid Rezatofighi. 2025. An empirical study on how video-llms answer video questions. arXiv preprint arXiv:2508.15360. — [arXiv:2508.15360](https://arxiv.org/abs/2508.15360) find
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, and 1 others. 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998.
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  15. Izzeddin Gür, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. 2023. Understanding html with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2803–2821. find
  16. Thai Quoc Hoang, Kung-Hsiang Huang, Shirley Kokane, Jianguo Zhang, Zuxin Liu, Ming Zhu, Jake Grigsby, Tian Lan, Michael S Ryoo, Chien-Sheng Wu, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. 2025. LAM SIMULATOR: Advancing data generation for large action model t… find
  17. Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. 2025. Tree search for llm agent reinforcement learning. arXiv preprint arXiv:2509.21240. — [arXiv:2509.21240](https://arxiv.org/abs/2509.21240) find
  18. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479. — [arXiv:2408.02479](https://arxiv.org/abs/2408.02479) find
  19. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  20. Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, and 1 others. 2025. Deepagent: A general reasoning agent with scalable toolsets. arXiv preprint arXiv:2510.21618. — [arXiv:2510.21618](https://arxiv.org/abs/2510.21618) find
  21. Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Xiang Zhang, and Suhang Wang. 2025. A comprehensive survey on reinforcement learning-based agentic search: Foundations, roles, optimizations, evaluations, and applications. arXiv preprint arXiv:2510.16724. — [arXiv:2510.16724](https://arxiv.org/abs/2510.16724) find
  22. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  23. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  24. Ziyu Ma, Chenhui Gou, Yiming Hu, Yong Wang, Bohan Zhuang, and Jianfei Cai. 2026. Where and what matters: Sensitivity-aware task vectors for many-shot multimodal in-context learning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(10):7892–7900. find
  25. Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. 2024. Drvideo: Document retrieval based long video understanding. arXiv preprint arXiv:2406.12846. — [arXiv:2406.12846](https://arxiv.org/abs/2406.12846) find
  26. Shinji Mai, Yunpeng Zhai, Ziqian Chen, Cheng Chen, Anni Zou, Shuchang Tao, Zhaoyang Liu, and Bolin Ding. 2025a. Cues: A curiosity-driven and environment-grounded synthesis framework for agentic rl. arXiv preprint arXiv:2512.01311. — [arXiv:2512.01311](https://arxiv.org/abs/2512.01311) find
  27. Xinji Mai, Haotian Xu, Zhong-Zhi Li, Xing W, Weinong Wang, Jian Hu, Yingying Zhang, and Wenqiang Zhang. 2025b. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773. — [arXiv:2505.07773](https://arxiv.org/abs/2505.07773) find
  28. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assis…
  29. Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan. 2025. Explorer: Scaling exploration-driven web trajectory synthesis for multimodal web agents. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6300–6323. find
  30. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. 2025. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  31. Qwen. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). find
  32. Qwen. 2024b. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  33. Qwen. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  34. Ram Ramrakhya, Andrew Szot, Omar Attia, Yuhao Yang, Anh Nguyen, Bogdan Mazoure, Zhe Gan, Harsh Agrawal, and Alexander Toshev. 2025. Scaling synthetic task generation for agents via exploration. arXiv preprint arXiv:2509.25047. — [arXiv:2509.25047](https://arxiv.org/abs/2509.25047) find
  35. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
  36. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  37. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations (ICLR).
  38. Chathurangi Shyalika, Ruwan Wickramarachchi, and Amit P Sheth. 2024. A comprehensive survey on rare event prediction. ACM Computing Surveys, 57(3):1– 39. find
  39. Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Agentbank: Towards generalized llm agents via fine-tuning on 50000+ interaction trajectories. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2124–2141. find
  40. Chuanneng Sun, Songjun Huang, and Dario Pompili. 2024. Llm-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2405.11106. — [arXiv:2405.11106](https://arxiv.org/abs/2405.11106) find
  41. Qwen Team. 2025. Qwen3-max: Just scale it. find
  42. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159. — [arXiv:1812.05159](https://arxiv.org/abs/1812.05159) find
  43. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Asso…
  44. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. find
  45. Hanlin Wang, Jian Wang, Chak Tou Leong, and Wenjie Li. 2025a. STeCa: Step-level trajectory calibration for LLM agent learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11597–11614, Vienna, Austria. Association for Computational Linguistics. find
  46. Yiming Wang, Da Yin, Yuedong Cui, Ruichen Zheng, Zhiqian Li, Zongyu Lin, Di Wu, Xueqing Wu, Chenchen Ye, Yu Zhou, and 1 others. 2025b. Llms as scalable, general-purpose simulators for evolving digital agent training. arXiv preprint arXiv:2510.14969. — [arXiv:2510.14969](https://arxiv.org/abs/2510.14969) find
  47. Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. 2025c. Co-evolving llm coder and unit tester via reinforcement learning. arXiv preprint arXiv:2506.03136. — [arXiv:2506.03136](https://arxiv.org/abs/2506.03136) find
  48. Yang Xiao, Mohan Jiang, Jie Sun, Keyu Li, Jifan Lin, Yumin Zhuang, Ji Zeng, Shijie Xia, Qishuo Hua, Xuefeng Li, and 1 others. 2025. Limi: Less is more for agency. arXiv preprint arXiv:2509.17567. — [arXiv:2509.17567](https://arxiv.org/abs/2509.17567) find
  49. Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. 2024. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09605. — [arXiv:2412.09605](https://arxiv.org/abs/2412.09605) find
  50. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  51. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  52. Junjie Ye, Xuanteng Gao, Keqing Zhang, Guodian Gao, Yutao Li, and Xiaozhi Liu. 2024. LLM-DA: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568. — [arXiv:2402.14568](https://arxiv.org/abs/2402.14568) find
  53. Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang. 2025. Demystifying reinforcement learning in agentic reasoning. arXiv preprint arXiv:2510.11701. — [arXiv:2510.11701](https://arxiv.org/abs/2510.11701) find
  54. Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. 2025. Agent-R: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425. — [arXiv:2501.11425](https://arxiv.org/abs/2501.11425) find
  55. Yunpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, and 1 others. 2025. Agentevolver: Towards efficient self-evolving agent system. arXiv preprint arXiv:2511.10395. — [arXiv:2511.10395](https://arxiv.org/abs/2511.10395) find
  56. Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, and 1 others. 2025a. Agent learning via early experience. arXiv preprint arXiv:2510.08558. find
  57. Shaolei Zhang, Ju Fan, Meihao Fan, Guoliang Li, and Xiaoyong Du. 2025b. Deepanalyze: Agentic large language models for autonomous data science. arXiv preprint arXiv:2510.16872. — [arXiv:2510.16872](https://arxiv.org/abs/2510.16872) find
  58. Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. 2025. Knowagent: Knowledge-augmented planning for llm-based agents. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3709–3732. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-coevolve-training-llm-agents-agent/