Elamur Research⌕
☰
home / papers / 2026 / A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains

A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains

Paper · 2026 · Xianren Zhang, Shreyas Prasad, Di Wang et al.

Summary

Existing e-commerce benchmarks for web agents (WebShop, WebVoyager, DeepShop and others) mostly test product search. They also only check whether a task was finished, not whether the agent caused damage along the way. This paper presents Amazon-Bench, built on the live Amazon.com site.

Queries come from a functionality-grounded pipeline with three stages:

  • crawling the website,
  • sampling pages so that their functions vary,
  • giving an LLM the simplified page content (AXTree) and interactive elements, so it writes requests the page can actually support. These requests are then rewritten and checked by humans.

The resulting tasks go well beyond search. They include managing addresses and wish lists, setting gift-card auto-reload, following brand stores, reading reviews and interacting with media.

An automated LLM-as-a-Judge framework scores each attempt from the query, screenshots and action log. It sorts every outcome into one of three classes: success, benign failure (nothing on the account changed) or harmful failure (an unwanted change to the account).

Tests of several LLM-based agents show they still struggle with complex requests and sometimes harm the user's account. This makes the benchmark useful for building safer shopping agents.

Key points

  • Task coverage: Amazon-Bench spans six task families: product search, deal search, account management, media interaction, store interaction and review checking. Compared baselines cover at most two of these.
  • Harmful failure rates: Overall rates range from 4.00% (Nova-Act) and 4.75% (GPT-4.1) up to 9.00% (GPT-4o). Product-interaction tasks are the riskiest, reaching 23.38% for GPT-4o.
  • Judge reliability: On 120 trajectories from GPT-4.1, Claude-3.7 and Deepseek-R1, the GPT-4o judge agreed with human annotators 92.5% of the time overall (90.0–95.0% per agent).
  • Efficiency: Deepseek-R1 had the slowest steps and used the most tokens. Claude-3.7 used few tokens but still had high latency.
  • Case studies and release:
    • Nova-Act kept scrolling for a save button that did not exist on the auto-reload page.
    • Claude-3.7 added a Coach bag to the cart twice, which counts as a harmful failure.
    • Code and data are released under the MIT license: github.com/Zood123/Amazon_Bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Xianren Zhang, Shreyas Prasad, Di Wang, Qiuhai Zeng, Suhang Wang, Wenbo Yan and Mat Hans. 2026. A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.68

@inproceedings{zhang2026functionality,
  title = {{A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains}},
  author = {Xianren Zhang, Shreyas Prasad, Di Wang, Qiuhai Zeng, Suhang Wang, Wenbo Yan and Mat Hans},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.68},
  url = {https://doi.org/10.18653/v1/2026.acl-long.68}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. 2024. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032. — [arXiv:2407.13032](https://arxiv.org/abs/2407.13032) find
  2. Amazon AGI Labs. 2025. Introducing amazon nova act. https://labs.amazon.science/blog/ nova-act. Amazon Science blog; research preview announcement. ↗
  3. Anthropic. 2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/ 3-5-models-and-computer-use. ↗
  4. Anthropic. 2025. Claude 3.7 sonnet system card. https://www.anthropic.com/news/ claude-3-7-sonnet. Accessed: 2025-10-04. ↗
  5. Sanxing Chen, Sam Wiseman, and Bhuwan Dhingra. 2024. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911. — [arXiv:2404.09911](https://arxiv.org/abs/2404.09911) find
  6. De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, and 1 others. 2024. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467. find
  7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114.
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024b. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  10. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computatio…
  11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  12. Xing Han Lu, Zdeněk Kasner, and Siva Reddy. 2024. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pages 33007–33056. PMLR.
  13. Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuying Chen. 2025. Deepshop: A benchmark for deep research shopping agents. arXiv preprint arXiv:2506.02839. — [arXiv:2506.02839](https://arxiv.org/abs/2506.02839) find
  14. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  15. Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350. — [arXiv:2503.23350](https://arxiv.org/abs/2503.23350) find
  16. OpenAI. 2025a. Introducing chatgpt agent: bridging research and action. https://openai.com/index/ introducing-chatgpt-agent/. ↗
  17. OpenAI. 2025b. Introducing deep research. https://openai.com/index/ introducing-deep-research/. ↗
  18. OpenAI. 2025c. Introducing GPT-4.1 model family. https://openai.com/index/gpt-4-1/. Accessed: 2025-07-09. ↗
  19. OpenAI. 2025d. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-10-04. ↗
  20. Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and 1 others. 2024. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373. — [arXiv:2406.12373](https://arxiv.org/abs/2406.12373) find
  21. Store Interaction: Follow the Coach store. • Review Check: Show 1-star reviews for Air-Pods Pro 2 from customers who did not purchase AppleCare+. • Media Interaction: Show all Kindle books by Kirsten Miller, sorted by price from low to high. find
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  23. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An illusion of progress? assessing the current state of web agents. URL https://arxiv.org/abs/2504.01382. ↗
  24. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  25. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.
  26. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-functionality-grounded-benchmark-evaluating-web/