Elamur Research⌕
☰
home / papers / 2026 / “Penny Wise, Pixel Foolish”: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

“Penny Wise, Pixel Foolish”: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

Paper · 2026 · Jiachen Qian and Zhaolu Kang

Summary

The paper examines how vulnerable screenshot-driven mobile shopping agents are when a user sets a price limit, for example "buy the cheapest". The authors name a failure mode Visual Dominance Hallucination (VDH). In VDH, tiny adversarial changes to a product image outweigh the price text that the agent has read correctly, so the agent clicks a more expensive item.

To exploit this, they propose PriceBlind, a white-box attack. A merchant perturbs only the pixels of their own product image. The attack combines two parts:

  • an action-targeted NLL loss on the coordinate tokens of the target click;
  • a Semantic-Decoupling Loss that pulls the item's CLIP embedding toward centroids of cheap-item anchors while keeping the original appearance.

To transfer the attack to closed models, they use Ensemble-DI-FGSM over Qwen-VL and LLaVA-1.6 as surrogates.

On their E-ShopBench benchmark, white-box attack success reaches about 80%, and transfer to commercial black-box models is still meaningful. Robust encoders reduce the attack only partly, while a Verify-then-Act pipeline brings success below 10%.

This matters because autonomous agents are starting to make real purchases. The authors note that their evaluation is a controlled, layout-aware setup, not a live deployment.

Key points

  • Benchmark: E-ShopBench has 200 scenarios with explicit price constraints. Perturbations are bounded by an L∞ budget of about 8/255.
  • White-box results: attack success rate (ASR) is around 80% on Mobile-Agent-v2 and AppAgent.
  • Black-box transfer: ASR is roughly 35–41% on GPT-4o, Gemini-1.5-Pro and Claude-3.5-Sonnet under a single-turn coordinate-selection protocol.
  • Attack design: the Semantic-Decoupling Loss relies on a Visual Anchor Bank of K = 500 generic low-cost item embeddings. The paper also gives a heuristic cross-attention argument for why perturbed visual keys can outweigh textual price tokens.
  • Defenses: Robust-CLIP and AdPO give only partial protection. Pairing Verify-then-Act with robust encoders drops ASR under 10%, and accuracy on clean inputs stays in the upper-80% range.
  • Limitations: generating one perturbation takes about 45 s on an A100. The attack transfers less well to non-CLIP encoders such as SigLIP and InternViT. All tests ran in a sandbox.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSoftware · 92 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Jiachen Qian and Zhaolu Kang. 2026. “Penny Wise, Pixel Foolish”: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.788

@inproceedings{qian2026penny,
  title = {{“Penny Wise, Pixel Foolish”: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations}},
  author = {Jiachen Qian and Zhaolu Kang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.788},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.788}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  2. Chiyu Chen, Xinhao Song, Yunkai Chai, Yang Yao, Haodong Zhao, Lijun Li, Jie Li, Yan Teng, Gongshen Liu, and Yingchun Wang. 2025. GhostEI-Bench: Do mobile agents resilience to environmental injection in dynamic on-device environments? arXiv preprint arXiv:2510.20333. — [arXiv:2510.20333](https://arxiv.org/abs/2510.20333) find
  3. Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML. find
  4. Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting adversarial attacks with momentum. In CVPR. find
  5. Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In CVPR. find
  6. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In ICLR.
  7. Qi Guo, Shanmin Pang, Xiaojun Jia, Yang Liu, and Qing Guo. 2025. Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models. IEEE Transactions on Information Forensics and Security. find
  8. Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, and James Bailey. 2025. X-Transfer attacks: Towards super transferable adversarial attacks on CLIP. In ICML. find
  9. Nikola Jovanović, Mislav Balunović, Dimitar Iliev Dimitrov, and Martin T. Vechev. 2023. FARE: Provably fair representation learning with practical certificates. In ICML. find
  10. Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2024. OmniACT: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In ECCV. find
  11. Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS. find
  12. Chaohu Liu, Tianyi Gui, Yu Liu, and Linli Xu. 2025a. AdPO: Enhancing the adversarial robustness of large vision-language models with preference optimization. arXiv preprint arXiv:2504.01735. — [arXiv:2504.01735](https://arxiv.org/abs/2504.01735) find
  13. Guohong Liu, Jialei Ye, Jiacheng Liu, Yuanchun Li, Wei Liu, Pengzhi Gao, Jian Luan, and Yunxin Liu. 2025b. Mobile GUI agents under real-world threats: Are we there yet? arXiv preprint arXiv:2507.04227. — [arXiv:2507.04227](https://arxiv.org/abs/2507.04227) find
  14. Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. CoCo-Agent: A comprehensive cognitive MLLM agent for smartphone GUI automation. In Findings of the Association for Computational Linguistics: ACL 2024. find
  15. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In ICLR.
  16. Ziqi Miao, Yi Ding, Lijun Li, and Jing Shao. 2025. Visual contextual attack: Jailbreaking MLLMs with image-driven context injection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9627–9644.
  17. Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. 2020. A self-supervised approach for adversarial robustness. In CVPR. find
  18. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  19. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2024. Visual adversarial examples jailbreak aligned large language models. In AAAI. find
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML.
  21. Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. 2024. Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In ICML. find
  22. Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. In Advances in Neural Information Processing Systems, pages 2686–2710. find
  23. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. In MobiCom. find
  24. Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. 2025. Dissecting adversarial robustness of multimodal LM agents. In ICLR. find
  25. Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. 2019. Improving transferability of adversarial examples with input diversity. In CVPR. find
  26. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441. find
  27. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In ICCV.
  28. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025a. AppAgent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 70:1– 70:20. find
  29. Tingwei Zhang, Rishi Jha, Eugene Bagdasarian, and Vitaly Shmatikov. 2024. Adversarial illusions in multimodal embeddings. In USENIX Security. find
  30. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025b. Attacking vision-language computer agents via pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8387–8401.
  31. Yitong Zhang, Ximo Li, Liyi Cai, and Jia Li. 2025c. Environmental injection attacks against GUI agents in realistic dynamic environments. arXiv preprint arXiv:2509.11250. — [arXiv:2509.11250](https://arxiv.org/abs/2509.11250) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2026-penny-wise-pixel-foolish-bypassing/