Elamur Research⌕
☰
home / papers / 2026 / Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data

Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data

Paper · 2026 · Yuxuan Lu, Jing Huang, Yan Han et al.

Summary

Studies of LLM agents that imitate people have mostly asked whether their behavior looks believable to human judges. They rarely check whether an agent reproduces what a real person actually does at each step. This paper measures that step-level accuracy, using online shopping as the test domain. The authors build ShopCART from real click-through logs of an e-commerce platform. Each session is a timeline of webpage observations paired with user actions such as searches, clicks, purchases or leaving the site. Given the session history and the current page, a model must produce the next action.

Prompt-only state-of-the-art models perform poorly on this task. Fine-tuning on the real logs helps considerably. Adding reasoning traces synthesized by Claude 3.5 Sonnet to the training data improves results further. The work offers the first rigorous, process-centric benchmark for this task. It shows that off-the-shelf agents should not be trusted as faithful stand-ins for real customers in uses such as A/B testing or usability studies.

Key points

  • Dataset scale: 31,865 shopping sessions from 3,526 users, with 230,965 actions in total. Sessions end in 4,432 purchases or 27,433 terminations.
  • Prompt-only baselines: DeepSeek-R1, Llama and Claude reach only 11.86% next-action accuracy.
  • Fine-tuned model: Qwen2.5-7B, fine-tuned with synthesized reasoning, reaches 17.26% action accuracy and 33.86% F1 on purchase prediction. These are gains of about 5.4 and 13.85 points over prompt-only baselines.
  • Availability: Data and code are released on Hugging Face as NEU-HAI/ShopCART.
  • Limitations:
    • The input is text-only simplified HTML, so visual cues are not captured.
    • The action space is limited to basic operations such as click and type.
    • The study covers a single domain.
    • The reasoning traces aim to improve prediction accuracy, not to reflect genuine human cognition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations🎓Northeastern UniversityBoston · United States57 works · 6,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Yaochen Xie, Yisi Sang, Qi He and Dakuo Wang. 2026. Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2034

@inproceedings{lu2026can,
  title = {{Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data}},
  author = {Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Yaochen Xie, Yisi Sang, Qi He and Dakuo Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2034},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2034}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, and Dakuo Wang. 2025. Multi-agent-as-judge: Aligning llm-agent-based automated evaluation with multi-dimensional human evaluation. arXiv preprint arXiv:2507.21028. find
  2. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  3. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  4. Vardhan Dongre, Xiaocheng Yang, Emre Can Acikgoz, Suvodip Dey, Gokhan Tur, and Dilek Hakkani-Tür. 2025. ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents. Preprint, arXiv:2411.00927. find
  5. David W. Eccles and Güler Arsal. 2017. The think aloud method: What is it and how do I use it? Qualitative Research in Sport, Exercise and Health, 9(4):514– 531. find
  6. Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. In The Twelfth International Conference on Learning Representations. find
  7. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computatio…
  8. Yuxuan Lu, Ting-Yao Hsu, Hansu Gu, Limeng Cui, Yaochen Xie, William P Headden III, Bingsheng Yao, Akash Veeragouni, Jiapeng Liu, Sreyashi Nag, et al. 2026. Agent a/b: Automated and scalable a/b testing on live websites with interactive llm agents. In Proceedings of the Extended Abstracts of the 2026… find
  9. Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Laurence Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. 2025. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. Preprint, arXiv:2502.12561. — [arXiv:2502.12561](https://arxiv.org/abs/2502.12561) find
  10. Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. 2024. RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation. Preprint, arXiv:2402.16667. find
  11. Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, and Giovanni Campagna. 2024. WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents. Preprint, arXiv:2404.05902. — [arXiv:2404.05902](https://arxiv.org/abs/2404.05902) find
  12. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, pages 1–22, New York,… find
  13. Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. Generative Agent Simulations of 1,000 People. Preprint, arXiv:2411.10109. find
  14. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. 2025. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. Preprint, arXiv:2411.02337. — [arXiv:2411.02337](https://arxiv.org/abs/2411.02337) find
  15. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. Preprint, arXiv:2307.07924.
  16. Lu Sun, Shihan Fu, Bingsheng Yao, Yuxuan Lu, Wenbo Li, Hansu Gu, Jiri Gesi, Jing Huang, Chen Luo, and Dakuo Wang. 2025. LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants? Preprint, arXiv:2509.21501. — [arXiv:2509.21501](https://arxiv.org/abs/2509.21501) find
  17. Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. AX-Nav: Replaying Accessibility Tests from Natural Language. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, pages 1–16, New York, NY, USA. Association for Compu… find
  18. Dakuo Wang, Ting-Yao Hsu, Yuxuan Lu, Hansu Gu, Limeng Cui, Yaochen Xie, William Headean, Bingsheng Yao, Akash Veeragouni, Jiapeng Liu, Sreyashi Nag, and Jessie Wang. 2025a. AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents. Preprint, arXiv:2504.09723. — [arXiv:2504.09723](https://arxiv.org/abs/2504.09723) find
  19. Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton, and Dakuo Wang. 2025b. OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Onl…
  20. Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip Torr, Bernard Ghanem, and Guohao Li. 2024. Can Large Language Model Agents Simulate Human Trust Behavior? Preprint, arXiv:2402.04559. — [arXiv:2402.04559](https://arxiv.org/abs/2402.04559) find
  21. Bingsheng Yao, Jiaju Chen, Chaoran Chen, April Yi Wang, Toby Jia-Jun Li, and Dakuo Wang. 2026. Through the lens of human-human collaboration: An configurable research platform for exploring human-agent collaboration. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pa… find
  22. Bingsheng Yao, Bo Sun, Yuanzhe Dong, Yuxuan Lu, and Dakuo Wang. 2025. Dprf: A generalizable dynamic persona refinement framework for optimizing behavior alignment between personalized llm role-playing agents and humans. arXiv preprint arXiv:2510.14205. — [arXiv:2510.14205](https://arxiv.org/abs/2510.14205) find
  23. Shunyu Yao, Howard Chen, John Yang, and Karthik R. Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems.
  24. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. Preprint, arXiv:2210.03629.
  25. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Shard… find
  26. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. Preprint, arXiv:2307.13854.
  27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Preprint, arXiv:2201.11903.
  28. Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. 2025. Collabllm: From passive responders to active collaborators. arXiv preprint arXiv:2502.00640. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2026-can-llm-agents-simulate-multi/