Elamur Research⌕
☰
home / papers / 2026 / OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation

OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation

Paper · 2026 · Ziyi Wang, Yuxuan Lu, Wenbo Li et al.

Summary

LLM agents can produce plausible web behavior, but it is hard to check whether they reproduce what a specific person would actually do. Public data that pairs real users' step-by-step actions with their reasons and profiles has been missing. The authors address this gap with OPeRA, an online shopping dataset built from real participants. A custom Chrome extension, ShoppingFlow, logged clicks, inputs, scrolls and page changes on Amazon together with HTML and screenshots. At decision points it occasionally asked users to explain their choice. A questionnaire and optional interviews supplied persona profiles covering demographics, shopping style and personality. Using this data, the authors define a benchmark: given a persona and the history of observations, actions and rationales, predict a user's next action and its rationale. They tested four LLMs, and all scored low on exact next-action accuracy. Including rationales helped most models, while personas mainly improved action-type classification. The models also tended to keep pushing toward a purchase instead of quitting a session early, as real users often do. OPeRA offers a grounded testbed for building personalized "digital twin" agents.

Key points

  • Dataset sizes:
    • OPeRA-full: 692 sessions from 51 users collected over four weeks, with 28,904 <action, observation> pairs and 604 user-provided rationales.
    • OPeRA-filtered: 527 sessions, 5,856 pairs and 207 rationales.
    • The dataset is released on Hugging Face as NEU-HAI/OPeRA.
  • Models and accuracy:
    • GPT-4.1, DeepSeek-R1, Claude-3.7 and Llama-3.3 were evaluated on OPeRA-test.
    • Exact next-action accuracy stayed low, for example about 10.75% for Claude-3.7 and 8.76% for Llama-3.3.
  • Input ablations:
    • Dropping historical rationales usually hurt performance, especially outcome prediction. Llama-3.3 was an exception.
    • Adding personas improved action-type and click-type prediction but not exact action accuracy.
  • Error analysis:
    • Clicking the wrong button accounted for roughly 61–76% of errors across models.
    • The terminate action was seldom predicted, with Claude and Llama never producing it.
  • Limitations:
    • The benchmark uses a reduced action space without scrolling or navigation.
    • It relies only on text and ignores the collected screenshots.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton and Dakuo Wang. 2026. OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2033

@inproceedings{wang2026opera,
  title = {{OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation}},
  author = {Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton and Dakuo Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2033},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2033}
}

References

68 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Q… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  2. Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find
  3. Anthropic. 2024. Claude 3.5 sonnet technical overview. https://www.anthropic.com/news/ claude-3-5-sonnet. https://www.anthropic. com/news/claude-3-5-sonnet. ↗
  4. Anthropic. 2025. Claude 3.7 sonnet and claude code. Accessed: 2025-04-06. find
  5. Živilė Baubonienė and Gintarė Gulevičiūtė. 2015. Ecommerce factors influencing consumers ‘online shopping decision. Social technologies, 5(1):62–73. find
  6. David Ben-Shimon, Alexander Tsikinovsky, Michael Friedmann, Bracha Shapira, Lior Rokach, and Johannes Hoerle. 2015. Recsys challenge 2015 and the yoochoose dataset. In Proceedings of the 9th ACM Conference on Recommender Systems, pages 357–358. find
  7. Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Yanfang Ye, Toby Jia-Jun Li, and Dakuo Wang. 2025a. Towards a design guideline for rpa evaluation: A survey of large language model-based role-playing agents. arXiv preprint arXiv:2502.13012. find
  8. Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. 2024. Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv preprint arXiv:2406.10819. — [arXiv:2406.10819](https://arxiv.org/abs/2406.10819) find
  9. Yiwei Chen, Soumyadeep Pal, Yimeng Zhang, Qing Qu, and Sijia Liu. 2025b. Unlearning isn’t invisible: Detecting unlearning traces in llms from model outputs. arXiv preprint arXiv:2506.14003. — [arXiv:2506.14003](https://arxiv.org/abs/2506.14003) find
  10. Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent V Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. 2024. Beyond demographics: aligning role-playing llm-based agents using human belief networks. arXiv preprint arXiv:2406.17232. find
  11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… find
  12. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  13. Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094.
  14. Lewis R Goldberg. 1992. The development of markers for the big-five factor structure. Psychological assessment, 4(1):26. find
  15. Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics, pages 148–170. find
  16. Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. In The Twelfth International Conference on Learning Representations. find
  17. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  18. Arief Helmi, Rita Komaladewi, Vita Sarasi, and Ledy Yolanda. 2023. Characterizing young consumer online shopping style: Indonesian evidence. Sustainability, 15(5):3988. find
  19. Jianwei Hou and Kevin Elliott. 2021. Mobile shopping intensity: Consumer demographics and motivations. Journal of Retailing and Consumer Services, 63:102741. find
  20. Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952. — [arXiv:2403.03952](https://arxiv.org/abs/2403.03952) find
  21. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. Soul: Unlocking the power of second-order optimization for llm unlearning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages…
  22. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516.
  23. Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu Han, Hanqing Lu, Zhengyang Wang, Ruirui Li, et al. 2024a. Amazon-m2: A multilingual multi-locale shopping session dataset for recommendation and text generation. Advances in Neural Information Processing Systems, 36. find
  24. Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, et al. 2024b. Shopping mmlu: A massive multi-task online shopping benchmark for large language models. arXiv preprint arXiv:2410.20745. — [arXiv:2410.20745](https://arxiv.org/abs/2410.20745) find
  25. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology. find
  26. Guimei Liu, Tam T Nguyen, Gang Zhao, Wei Zha, Jianbo Yang, Jianneng Cao, Min Wu, Peilin Zhao, and Wei Chen. 2016. Repeat buyer prediction for e-commerce. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 155–164. find
  27. Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. 2024. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820. — [arXiv:2411.00820](https://arxiv.org/abs/2411.00820) find
  28. Ryan Louie, Ifdita Hasan Orney, Juan Pablo Pacheco, Raj Sanjay Shah, Emma Brunskill, and Diyi Yang. 2025. Can llm-simulated practice and feedback up-skill human counselors? a randomized study with 90+ novice counselors. Preprint, arXiv:2505.02428. find
  29. Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Jiri Gesi, Yaochen Xie, Zheshen, Wang, Qi He, and Dakuo Wang. 2025a. Prompting is not all you need! evaluating llm agent simulation methodologies with real-world online customer behavior data. Preprint, arXiv:2503.20749. — [arXiv:2503.20749](https://arxiv.org/abs/2503.20749) find
  30. Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Laurence Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. 2025b. Uxagent: An llm agent-based usability testing framework for web design. Preprint, arXiv:2502.12561. — [arXiv:2502.12561](https://arxiv.org/abs/2502.12561) find
  31. Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941. find
  32. Meta. 2024. Llama-3.3-70b-instruct. find
  33. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983.
  34. Daniel Mican and Dan-Andrei Sitar-Taut. 2020. Analysis of the factors impacting the online shopping decision-making process. Studia Universitatis Babes-Bolyai, 65(1):54–66. find
  35. Isabel Briggs Myers, Mary H. McCaulley, Naomi L. Quenk, and Allen L. Hammer. 1998. MBTI Manual: A Guide to the Development and Use of the Myers-Briggs Type Indicator, 3rd edition. Consulting Psychologists Press, Palo Alto, CA. find
  36. Tahmid Nayeem and Jean Marie-IpSooching. 2022. Revisiting sproles and kendall’s consumer styles inventory (csi) in the 21st century: A case of australian consumers decision-making styles in the context of high and low-involvement purchases. Marketing, 7(2):7–17. find
  37. Nurul Zarirah Nizam and Jaafar Abdullah Jaafar. 2018. Interactive online advertising: The effectiveness of marketing strategy towards customers purchase decision. International Journal of Human and Technology Interaction (IJHaTI), 2(2):9–16. find
  38. OpenAI. 2025b. Introducing operator. find
  39. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  40. Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. Generative Agent Simulations of 1,000 People. Preprint, arXiv:2411.10109. find
  41. Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning. 2024. ecellm: Generalizing large language models for e-commerce from large-scale, high-quality instruction data. arXiv preprint arXiv:2402.08831. — [arXiv:2402.08831](https://arxiv.org/abs/2402.08831) find
  42. Gyan Prakash, Pankaj Kumar Singh, and Rambalak Yadav. 2018. Application of consumer style inventory (csi) to predict young indian consumer’s intention to purchase organic food products. Food quality and preference, 68:90–97. find
  43. Jing-bo Shao, Zhen-zhen Li, and Ming-ye Hu. 2014. The impact of online reviews on consumers’ purchase decisions in online shopping. In 2014 International Conference on Management Science & Engineering 21th Annual Conference Proceedings, pages 287–293. IEEE. find
  44. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158.
  45. Yimin Shi, Yang Fei, Shiqi Zhang, Haixun Wang, and Xiaokui Xiao. 2025. You are what you bought: Generating customer personas for e-commerce applications. arXiv preprint arXiv:2504.17304. — [arXiv:2504.17304](https://arxiv.org/abs/2504.17304) find
  46. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  47. Mastercard SpendingPulse. 2023. Holiday spending 2023. Accessed: March 5, 2025. find
  48. George B Sprotles and Elizabeth L Kendall. 1986. A methodology for profiling consumers’ decision-making styles. Journal of Consumer Affairs, 20(2):267–279. find
  49. Karthik Sreedhar, Alice Cai, Jenny Ma, Jeffrey V Nickerson, and Lydia B Chilton. 2025. Simulating cooperative prosocial behavior with multi-agent llms: Evidence and mechanisms for ai agents to inform policy decisions. In Proceedings of the 30th International Conference on Intelligent User Interfaces… find
  50. Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. AX-Nav: Replaying Accessibility Tests from Natural Language. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, pages 1–16, New York, NY, USA. Association for Compu… find
  51. Dakuo Wang, Ting-Yao Hsu, Yuxuan Lu, Limeng Cui, Yaochen Xie, William Headean, Bingsheng Yao, Akash Veeragouni, Jiapeng Liu, Sreyashi Nag, et al. 2025a. Agenta/b: Automated and scalable web a/btesting with interactive llm agents. arXiv preprint arXiv:2504.09723. — [arXiv:2504.09723](https://arxiv.org/abs/2504.09723) find
  52. Lei Wang, Jingsen Zhang, Hao Yang, Zhiyuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. 2024a. User behavior simulation with large language model based agents. Preprint, arXiv:2306.02552. find
  53. Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. 2024b. Sotopia-π: Interactive learning of socially intelligent language agents. arXiv preprint arXiv:2403.08715. — [arXiv:2403.08715](https://arxiv.org/abs/2403.08715) find
  54. Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, and Yingzhen Yang. 2023. Recmind: Large language model powered agent for recommendation. arXiv preprint arXiv:2308.14296. find
  55. Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025b. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733. — [arXiv:2501.11733](https://arxiv.org/abs/2501.11733) find
  56. Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Yisi Sang, Hanqing Lu, Manling Li, et al. 2026. Trajectory2task: Training robust tool-calling agents with synthesized yet verifiable data for complex user intents. arXiv preprint arXiv:2601.20144.
  57. Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, and Dakuo Wang. 2025c. Customer-r1: Personalized simulation of human behaviors via rl-based llm agent in online shopping. arXiv preprint arXiv:2510.07230. — [arXiv:2510.07230](https://arxiv.org/abs/2510.07230) find
  58. Liying Wei. 2016. Decision-making behaviours toward online shopping. International Journal of Marketing Studies, 8(3):111–121. find
  59. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155.
  60. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622. find
  61. Shunyu Yao, Howard Chen, John Yang, and Karthik R. Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents.
  62. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  63. Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. 2024. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592. — [arXiv:2402.11592](https://arxiv.org/abs/2402.11592) find
  64. Yimeng Zhang, Jiri Gesi, Ran Xue, Tian Wang, Ziyi Wang, Yuxuan Lu, Sinong Zhan, Huimin Zeng, Qingjun Cui, Yufan Guo, et al. 2025a. See, think, act: Online shopper behavior simulation with vlm agents. arXiv preprint arXiv:2510.19245. — [arXiv:2510.19245](https://arxiv.org/abs/2510.19245) find
  65. Yimeng Zhang, Tian Wang, Jiri Gesi, Ziyi Wang, Yuxuan Lu, Jiacheng Lin, Sinong Zhan, Vianne Gao, Ruochen Jiao, Junze Liu, et al. 2025b. Shop-r1: Rewarding llms to simulate human behavior in online shopping via reinforcement learning. arXiv preprint arXiv:2507.17842. — [arXiv:2507.17842](https://arxiv.org/abs/2507.17842) find
  66. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024a. WebArena: A Realistic Web Environment for Building Autonomous Agents. Preprint, arXiv:2307.13854.
  67. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024b. Webarena: A realistic web environment for building autonomous agents. Preprint, arXiv:2307.13854.
  68. Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. 2018. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1079–1088. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-opera-dataset-observation-persona-rationale/