Elamur Research⌕
☰
home / papers / 2026 / Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces

Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces

Paper · 2026 · Yilin Zhang, Yingkai Hua, Chunyu Wei et al.

Summary

Web agents built on vision-language models often click on manipulative interface elements such as fake download buttons, misleading ads, and disguised pop-ups. The authors report that leading GUI agents are fooled by these dark patterns more than 70% of the time. Earlier work either detects such patterns separately from the agent's task or describes attacks without offering defenses. This paper frames deception-aware defense as its own problem and introduces DUDE (Deceptive UI Detector & Evaluator). DUDE is a plug-in evaluator that checks each click an agent proposes before it is executed. It is built in two stages:

  • Reinforcement learning stage: the evaluator is trained with a hybrid reward that penalizes approving a deceptive click much more heavily than wrongly rejecting a legitimate one.
  • Experience summarization stage: collected failures are condensed into short prompt guidance, so the system keeps improving without further weight updates.

To evaluate DUDE, the authors release RUC (Real UI Clickboxes), a benchmark of annotated scenarios. DUDE cuts deception susceptibility by 53.8% while keeping task performance intact. It also reduces the number of interaction steps, which makes it a practical safety layer for deploying web agents.

Key points

  • RUC benchmark: 1,407 scenarios across four domains and several deception categories. Each scenario has bounding boxes for the correct target and the deceptive one, and every click is labeled as legitimate (1), ineffective (0), or deceptive (−1).
  • Reward design:
    • Approving a deceptive click carries a severity weight of 10.
    • Mistakes on ineffective clicks are weighted 1+β, where β depends on the size of the predicted region.
    • A confidence scalar α eases penalties for clicks near region boundaries.
  • Ablation: removing the severity weighting raises the Fatal Error Rate from 9.75% to 27.53%, and removing the confidence adjustment raises it to 17.25%.
  • Efficiency on UI-TARS-1.5-7B:
    • Average steps drop from 17.65 to 3.58.
    • Total wall-clock time falls from 217.62s to 48.47s.
    • The Stage-2 guidance adds only about 200–400 tokens per call.
  • Transfer and limitations: applying the learned Stage-2 prompt to a closed-source model without any extra tuning raises SR from 54.12% to 62.50%. The evaluation is limited to static, English-only screenshots.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations🏢Ant GroupHangzhou · China51 works · 1,211 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 works

Cite

Cite as Yilin Zhang, Yingkai Hua, Chunyu Wei, Xin Wang and Yueguo Chen. 2026. Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.310

@inproceedings{zhang2026don,
  title = {{Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces}},
  author = {Yilin Zhang, Yingkai Hua, Chunyu Wei, Xin Wang and Yueguo Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.310},
  url = {https://doi.org/10.18653/v1/2026.acl-long.310}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mathieu Andreux, Breno Baldas Skuk, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Pierre-Louis Cedoz, Antoine Chassang, Mickaël Chen, Alexandra D. Constantinou, Antoine d’Andigné, Hubert de La Jonquière, Aurélien Delfosse, Ludovic Denoyer, Alexis Deprez,… — [arXiv:2506.02865](https://arxiv.org/abs/2506.02865) find
  2. Jieshan Chen, Jiamou Sun, Sidong Feng, Zhenchang Xing, Qinghua Lu, Xiwei Xu, and Chunyang Chen. 2023. Unveiling the tricks: Automated detection of dark patterns in mobile applications. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 2023, San Francisco… find
  3. Yurun Chen, Xueyu Hu, Keting Yin, Juncheng Li, and Shengyu Zhang. 2025. AEIA-MN: evaluating the robustness of multimodal llm-powered mobile agents against active environmental injection attacks. CoRR, abs/2502.13053. — [arXiv:2502.13053](https://arxiv.org/abs/2502.13053) find
  4. Phil Cuvin, Hao Zhu, and Diyi Yang. 2026. How dark patterns manipulate web agents. In The Fourteenth International Conference on Learning Representations. find
  5. Devin Ersoy, Brandon Lee, Ananth Shreekumar, Arjun Arunasalam, Muhammad Ibrahim, Antonio Bianchi, and Z. Berkay Celik. 2025. Investigating the impact of dark patterns on llm-based web agents. CoRR, abs/2510.18113. — [arXiv:2510.18113](https://arxiv.org/abs/2510.18113) find
  6. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. O…
  7. Colin M. Gray, Yubo Kou, Bryan Battles, Joseph Hog-gatt, and Austin L. Toombs. 2018. The dark (patterns) side of UX design. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, Canada, April 21-26, 2018, page 534. ACM. find
  8. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. Cogagent: A visual language model for GUI agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 20…
  9. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Comp…
  10. Esben Kran, Jord Nguyen, Akash Kundu, Sami Jawhar, Jinsuk Park, and Mateusz Maria Jurewicz. 2025. Darkbench: Benchmarking dark patterns in large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  11. Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. 2025. Os-harm: A benchmark for measuring safety of computer use agents. CoRR, abs/2506.14866. — [arXiv:2506.14866](https://arxiv.org/abs/2506.14866) find
  12. Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2025. Ferret-ui 2: Mastering universal user interface understanding across platforms. In The Thirteenth International Conference on Learning Representati… find
  13. Zeyi Liao, Jaylen Jones, Linxi Jiang, Eric Fosler-Lussier, Yu Su, Zhiqiang Lin, and Huan Sun. 2025. Redteamcua: Realistic adversarial testing of computer-use agents in hybrid web-os environments. CoRR, abs/2505.21936. — [arXiv:2505.21936](https://arxiv.org/abs/2505.21936) find
  14. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. Showui: One vision-language-action model for GUI visual agent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA,… find
  15. Arunesh Mathur, Gunes Acar, Michael Friedman, Elena Lucherini, Jonathan R. Mayer, Marshini Chetty, and Arvind Narayanan. 2019. Dark patterns at scale: Findings from a crawl of 11k shopping websites. Proc. ACM Hum. Comput. Interact., 3(CSCW):81:1– 81:32. find
  16. Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. 2024. Fight back against jailbreaking via prompt adversarial tuning. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 202… find
  17. Arpit Narechania, Kaustubh Odak, Mennatallah El-Assady, and Alex Endert. 2025. Provenancewidgets: A library of UI control elements to track and dynamically overlay analytic provenance. IEEE Trans. Vis. Comput. Graph., 31(1):1235–1245. find
  18. Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. find
  19. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent Q: advanced reasoning and learning for autonomous AI agents. CoRR, abs/2408.07199.
  20. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  21. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025. UI-TARS: pioneering automated GUI interaction with…
  22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.
  23. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of M… find
  24. Zewei Shi, Ruoxi Sun, Jieshan Chen, Jiamou Sun, Minhui Xue, Yansong Gao, Feng Liu, and Xingliang Yuan. 2025. 50 shades of deceptive patterns: A unified taxonomy, multimodal detection, and security implications. In Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 Ap… find
  25. Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi N. Georgiev, Rocktim Jyoti Das, and Preslav Nakov. 2024. Factuality of large language models: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 1…
  26. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  27. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, and 80 others. 2025. GLM-4.5: agentic, reasoning, and coding (ARC) fou…
  28. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025. Attacking vision-language computer agents via pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 -August 1, 2025, pages 8387–8401. Association for C…
  29. Zhuosheng Zhang and Aston Zhang. 2024. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 3132–3149. Association for Computational Linguistics.
  30. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representation…
  31. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2024. OS-ATLAS: A foundation action model for generalist GUI agents. CoRR, abs/2410.23218.
  32. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-don-t-click-that-teaching/