Elamur Research⌕
☰
home / papers / 2025 / APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

Paper · 2025 · Akshara Prabhakar, Zuxin Liu, Ming Zhu et al.

Summary

Building capable multi-turn AI agents requires training data that captures realistic exchanges between humans and agents. Such data is rare and costly to collect by hand. APIGen-MT, from Salesforce AI Research, addresses this with a two-stage synthetic pipeline. First, an LLM-driven generator creates task "blueprints." Each blueprint pairs a user intent with ground-truth API actions and expected outputs, built from sampled APIs, domain rules, data and user personas. Every blueprint must pass format and execution checks and a majority vote by an LLM reviewer committee, and failures are sent back with summarized feedback for revision. Second, each validated blueprint guides a simulated conversation in which an LLM acts as the human user while the agent under test works inside an environment where its actions are actually run. Only trajectories whose actions match the ground truth are kept. The authors fine-tuned Llama and Qwen models on this data to create the xLAM-2-fc-r family. These models beat open baselines and rival or surpass GPT-4o and Claude 3.5 on τ-bench and BFCL v3. Smaller models perform especially well, which shows that verified synthetic data can substitute for expensive manual annotation.

Key points

  • Released resources: 5K trajectories (APIGen-MT-5k) and the xLAM-2-fc-r models, published on Hugging Face (Salesforce/APIGen-MT-5k, Salesforce/xLAM-2).
  • Model family: Models range from 1B to 70B parameters and are built on Llama 3.1/3.2 and Qwen 2.5.
  • Task modeling and complexity: The authors frame multi-turn interaction as a POMDP. They use "reverse task recombination" to make tasks more complex.
  • Results on τ-retail: Over 5 trials, xLAM-2-70b-fc-r reaches a 67.1% average success rate, compared with 62.8% for gpt-4o.
    • A Best-of-N user simulator raises scores to 68.8% and 67.0% respectively.
    • It also cuts variance: from 9.7 to 4.0 for xLAM and from 11.1 to 2.6 for gpt-4o.
  • Consistency on τ-airline: xLAM-2-70b-fc-r scores higher than Claude on pass^5 despite a slightly lower pass^1, indicating more reliable behavior across trials.
  • Limitations: The pipeline discards failed trajectories and its validation adds compute overhead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢SalesforceSan Francisco · United States59 works · 18,172 citations🏢OpenAISan Francisco · United States23 works · 27,874 citations🎓Westlake UniversityHangzhou · China51 works · 8,572 citations🎓Emory UniversityAtlanta · United States42 works · 39,899 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,105 works

Cite

Cite as Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese and Caiming Xiong. 2025. APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. arXiv:2504.03601.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.03601 · arXiv 2504.03601

@misc{prabhakar2025apigen,
  title = {{APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay}},
  author = {Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese and Caiming Xiong},
  year = {2025},
  doi = {10.48550/arxiv.2504.03601},
  eprint = {2504.03601},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.03601}
}

References

56 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Agashe, J. Han, S. Gan, J. Yang, A. Li, and X. E. Wang. Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164, 2024. — [arXiv:2410.08164](https://arxiv.org/abs/2410.08164) find
  2. Anthropic. Claude 3.5 sonnet, 2024. 3-5-models-and-computer-use. URL https://www.anthropic.com/news/Claude 3.7 sonnet, 2025. ↗
  3. Anthropic. claude-3-7-sonnet. URL https://www.anthropic.com/news/ ↗
  4. Anthropic. Claude think tool, 2025. URL https://www.anthropic.com/engineering/ claude-think-tool. ↗
  5. A. Antoniades, A. Örwall, K. Zhang, Y. Xie, A. Goyal, and W. Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285, 2024. find
  6. S. Arcadinho, D. Aparício, and M. Almeida. Automated test generation to evaluate tool-augmented llms as conversational ai agents. arXiv preprint arXiv:2409.15934, 2024. find
  7. D. Bahdanau, N. Gontier, G. Huang, E. Kamalloo, R. Pardinas, A. Piché, T. Scholak, O. Shliazhko, J. P. Tremblay, K. Ghanem, et al. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint arXiv:2412.08445, 2024. find
  8. Z. Bi, K. Han, C. Liu, Y. Tang, and Y. Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078, 2024.
  9. T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
  10. CAMEL-AI.org. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. https://github.com/camel-ai/owl, 2025. Accessed: 2025-03-07. ↗
  11. M. Chen, sunhaoze, T. Li, F. Yang, H. Liang, KeerLu, B. CUI, W. Zhang, Z. Zhou, and weipeng chen. Facilitating multi-turn function calling for LLMs via compositional instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=… ↗
  12. Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.12881, 2024.
  13. Z. Chen, M. Li, Y. Huang, Y. Du, M. Fang, and T. Zhou. Atlas: Agent tuning via learning critical steps. arXiv preprint arXiv:2503.02197, 2025. find
  14. S. Cognition. Apt-1 blog, 2025. URL https://www.scaledcognition.com/blog/apt-1. ↗
  15. T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691. ↗
  16. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  17. T. Ge, X. Chan, X. Wang, D. Yu, H. Mi, and D. Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https://arxiv.org/abs/2406.20094. ↗
  18. S. A. Hayati, T. Jung, T. Bodding-Long, S. Kar, A. Sethy, J.-K. Kim, and D. Kang. Chain-of-instructions: Compositional instruction tuning on large language models. arXiv preprint arXiv:2402.11532, 2024. — [arXiv:2402.11532](https://arxiv.org/abs/2402.11532) find
  19. K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments, 2025. URL https://arxiv.org/abs/2411.02305. ↗
  20. E. Levi and I. Kadar. Intellagent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.11067, 2025. — [arXiv:2501.11067](https://arxiv.org/abs/2501.11067) find
  21. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  22. X. Li, H. Zou, and P. Liu. Torl: Scaling tool-integrated rl, 2025. URL https://arxiv.org/ abs/2503.23383. ↗
  23. Y. Li, Y. Li, X. Wang, Y. Jiang, Z. Zhang, X. Zheng, H. Wang, H.-T. Zheng, P. Xie, P. S. Yu, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arXiv:2411.02937, 2024. — [arXiv:2411.02937](https://arxiv.org/abs/2411.02937) find
  24. W. Liu, X. Huang, X. Zeng, X. Hao, S. Yu, D. Li, S. Wang, W. Gan, Z. Liu, Y. Yu, et al. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920, 2024.
  25. Z. Liu, J. Zhang, K. Asadi, Y. Liu, D. Zhao, S. Sabach, and R. Fakoor. Tail: Task-specific adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.05905, 2023. — [arXiv:2310.05905](https://arxiv.org/abs/2310.05905) find
  26. Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, J. Tan, W. Yao, Z. Liu, Y. Feng, R. RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463–54482, 2024. find
  27. I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017. Decoupled weight decay regularization. arXiv preprint find
  28. J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682, 2024.
  29. A. Mitra, S. Patel, T. Chakrabarty, and C. Baral. Agentinstruct: An agentic framework for generating high-quality synthetic instruction data. arXiv preprint arXiv:2402.12360, 2024. — [arXiv:2402.12360](https://arxiv.org/abs/2402.12360) find
  30. J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024.
  31. J. Pan, R. Shar, J. Pfau, A. Talwalkar, H. He, and V. Chen. When benchmarks talk: Re-evaluating code llms with interactive feedback, 2025. URL https://arxiv.org/abs/2502.18413. ↗
  32. J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. find
  33. Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(4):1–40, 2024.
  34. Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan,… find
  35. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA, 202… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) find
  36. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023.
  37. S. Sengupta, K. Curtis, A. Mallipeddi, A. Mathur, J. Ross, and L. Gou. Mag-v: A multi-agent framework for synthetic data generation and verification. arXiv preprint arXiv:2412.04494, 2024. — [arXiv:2412.04494](https://arxiv.org/abs/2412.04494) find
  38. J. Shim, G. Seo, C. Lim, and Y. Jo. Tooldial: Multi-turn dialogue generation method for tool-augmented language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=J1J5eGJsKZ. ↗
  39. N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=vAElhFcKW6. ↗
  40. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. find
  41. H. Su, R. Sun, J. Yoon, P. Yin, T. Yu, and S. Ö. Arık. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025. — [arXiv:2501.10893](https://arxiv.org/abs/2501.10893) find
  42. S. Tang, X. Pang, Z. Liu, B. Tang, R. Ye, X. Dong, Y. Wang, and S. Chen. Synthesizing post-training data for llms through multi-agent simulation. arXiv preprint arXiv:2410.14251, 2024.
  43. J. Wang, J. Zhou, M. Wen, X. Mo, H. Zhang, Q. Lin, C. Jin, X. Wang, W. Zhang, and Q. Peng. Hammerbench: Fine-grained function-calling evaluation in real mobile device scenarios. arXiv preprint arXiv:2412.16516, 2024. — [arXiv:2412.16516](https://arxiv.org/abs/2412.16516) find
  44. G. Wölflein, D. Ferber, D. Truhn, O. Arandjelović, and J. N. Kather. Llm agents making agent tools. arXiv preprint arXiv:2502.11705, 2025.
  45. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. 2024. find
  46. J. Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop @ NeurIPS’23, 2023. URL https://openreview.net/forum?id=KOZwk7BFc3. ↗
  47. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024.
  48. R. Yang, F. Ye, J. Li, S. Yuan, Y. Zhang, Z. Tu, X. Li, and D. Yang. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024, 2025. — [arXiv:2503.16024](https://arxiv.org/abs/2503.16024) find
  49. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.
  50. F. Yin, Z. Wang, I.-H. Hsu, J. Yan, K. Jiang, Y. Chen, J. Gu, L. T. Le, K.-W. Chang, C.-Y. Lee, H. Palangi, and T. Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation, 2025. URL https://arxiv.org/abs/2503.07826. ↗
  51. Y. Zeng, X. Ding, Y. Wang, W. Liu, W. Ning, Y. Hou, X. Huang, B. Qin, and T. Liu. Boosting tool use of large language models via iterative reinforced fine-tuning. arXiv preprint arXiv:2501.09766, 2025. — [arXiv:2501.09766](https://arxiv.org/abs/2501.09766) find
  52. J. Zhang, T. Lan, M. Zhu, Z. Liu, T. Hoang, S. Kokane, W. Yao, J. Tan, A. Prabhakar, H. Chen, et al. xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215, 2024.
  53. J. Zhang, T. Hoang, M. Zhu, Z. Liu, S. Wang, T. Awalgaonkar, A. Prabhakar, H. Chen, W. Yao, Z. Liu, et al. Actionstudio: A lightweight framework for data and training of action models. arXiv preprint arXiv:2503.22673, 2025. — [arXiv:2503.22673](https://arxiv.org/abs/2503.22673) find
  54. K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. Rithesh, T. Lan, L. Li, R. Lou, J. Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. In The Thirteenth International Conference on Learning Representations, 2024. find
  55. Y. Zhang, J. Lu, and N. Jaitly. Probing the multi-turn planning capabilities of LLMs via 20 question games. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495–1516, Bangkok, Th… find
  56. Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prabhakar-2025-apigen-mt-agentic-pipeline-multi/