Elamur Research⌕
☰
home / papers / 2026 / Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models

Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models

Paper · 2026 · Jiaxin Ye, Weihai Li, Ying Wang et al.

Summary

Vision-language pre-trained (VLP) models such as CLIP, BLIP and ALBEF can be fooled by adversarial inputs. Universal adversarial perturbations (UAPs), where one perturbation works for any input, are cheap to deploy, but existing multimodal UAPs transfer poorly to unseen (black-box) models. The authors argue the cause is that current objectives treat image and text the same way, or match whole distributions, and ignore how the two modalities relate to each other. They propose ARG-Attack, an optimization-based method with no generator network. It learns a single image perturbation and a single text perturbation using three losses. A cosine loss pushes visual features off their meaning in a chosen direction. A center shift loss pulls adversarial embeddings toward a common batch center. A relational polarity loss breaks the similarity ranking between matched and mismatched image–text pairs. An Adam-style gradient aggregation step reduces gradient conflicts across samples and the error from sign-based updates. In white-box tests the method is roughly on par with the state-of-the-art CPGC. It clearly beats CPGC in black-box transfer and needs much less training. This makes it useful for robustness auditing.

Key points

  • Losses and optimizer: The objective combines L_cos, the center shift loss (L_CS) and the relational polarity loss (L_RPL). Optimization uses Adam-inspired adaptive gradient correction.
  • Efficiency: Training cost is reported as up to 30× lower than the generator-based CPGC (C-PGC).
  • Image captioning on MSCOCO (victim: BLIP):
    • Clean BLIP scores 39.7 BLEU@4 and 133.3 CIDEr.
    • Perturbations crafted on TCL lower these to 26.3 and 87.8, relative drops of about 33% and 34%.
    • Perturbations crafted on BLIP itself give 24.5 BLEU@4 and 15.6 SPICE.
  • Ablation: Tests use Flickr30K, with ALBEF as the surrogate and CLIPcnn as the target. Removing any one loss term lowers transfer. The center shift loss mainly keeps geometry stable, and RPL mainly breaks cross-modal associations.
  • Coverage and limitation: Evaluation spans image-text retrieval and captioning, with visual grounding and cross-dataset transfer in the appendices. White-box results do not beat CPGC on every model and metric.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations🎓University of Science and Technology of ChinaHefei · China145 works · 114,409 citations🎓Northeastern UniversityShenyang · China53 works · 2,205 citations🎓Beijing University of TechnologyBeijing · China16 works · 1,910 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Jiaxin Ye, Weihai Li, Ying Wang, Simeng Qin, Zhitao Zeng and Zikai Xu. 2026. Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.393

@inproceedings{ye2026asymmetric,
  title = {{Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models}},
  author = {Jiaxin Ye, Weihai Li, Ying Wang, Simeng Qin, Zhitao Zeng and Zikai Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.393},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.393}
}

References

53 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer.
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  3. Junyoung Byun, Seungju Cho, Myung-Joon Kwon, Hee-Seon Kim, and Changick Kim. 2022. Improving the transferability of targeted adversarial examples through object-based diverse input. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15244–15253. find
  4. Bin Chen, Jiali Yin, Shukai Chen, Bohao Chen, and Ximeng Liu. 2023. An adaptive model ensemble adversarial attack for boosting adversarial transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4489–4498. find
  5. Yaya Cheng, Jingkuan Song, Xiaosu Zhu, Qilong Zhang, Lianli Gao, and Heng Tao Shen. 2021. Fast gradient non-sign methods. arXiv preprint arXiv:2110.12734. — [arXiv:2110.12734](https://arxiv.org/abs/2110.12734) find
  6. Kenneth T Co, Luis Muñoz-González, Leslie Kanthan, Ben Glocker, and Emil C Lupu. 2021. Universal adversarial robustness of texture and shape-biased models. In 2021 IEEE International Conference on Image Processing (ICIP), pages 799–803. IEEE. find
  7. Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1625–1… find
  8. Hao Fang, Jiawei Kong, Wenbo Yu, Bin Chen, Jiawei Li, Hao Wu, Shutao Xia, and Ke Xu. 2024. One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models. arXiv preprint arXiv:2406.05491. — [arXiv:2406.05491](https://arxiv.org/abs/2406.05491) find
  9. Jiyuan Fu, Zhaoyu Chen, Kaixun Jiang, Haijing Guo, Jiafeng Wang, Shuyong Gao, and Wenqiang Zhang. 2024. Improving adversarial transferability of vision-language pre-training models through collaborative multimodal interaction. arXiv preprint arXiv:2403.10883. — [arXiv:2403.10883](https://arxiv.org/abs/2403.10883) find
  10. Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, and Qing Guo. 2024. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. In European Conference on Computer Vision, pages 442– 460. Springer. find
  11. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  12. Jindong Gu, Xiaojun Jia, Pau de Jorge, Wenqain Yu, Xinwei Liu, Avery Ma, Yuan Xun, Anjun Hu, Ashkan Khakzar, Zhijiang Li, and 1 others. 2023. A survey on transferability of adversarial examples across deep neural networks. arXiv preprint arXiv:2310.17626. — [arXiv:2310.17626](https://arxiv.org/abs/2310.17626) find
  13. Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849. find
  14. Valentin Khrulkov and Ivan Oseledets. 2018. Art of singular vectors and universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8562–8570. find
  15. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pages 5583–5594. PMLR.
  16. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.
  17. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705. find
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  19. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pag…
  20. Xuannan Liu, Yaoyao Zhong, Yuhang Zhang, Lixiong Qin, and Weihong Deng. 2023. Enhancing generalization of universal adversarial perturbation through gradient aggregation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4435–4444. find
  21. Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. 2023. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 102–111. find
  22. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
  23. Xiaofeng Mao, Yuefeng Chen, Yuhong Li, Yuan He, and Hui Xue. 2020. Gap++: Learning to generate target-conditioned adversarial examples. arXiv preprint arXiv:2006.05097. — [arXiv:2006.05097](https://arxiv.org/abs/2006.05097) find
  24. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765–1773. find
  25. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582. find
  26. Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. 2018. Nag: Network for adversary generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 742–751. find
  27. Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. 2021. On improving adversarial transferability of vision transformers. arXiv preprint arXiv:2106.04169. — [arXiv:2106.04169](https://arxiv.org/abs/2106.04169) find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  29. Huy Phan, Yi Xie, Siyu Liao, Jie Chen, and Bo Yuan. 2020. Cag: A real-time low-cost enhanced-robustness high-transferability content-aware adversarial attack generator. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5412–5419. find
  30. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649… find
  31. Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. 2018. Generative adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4422–4431. find
  32. Yaguan Qian, Shuke He, Chenyu Zhao, Jiaqiang Sha, Wei Wang, and Bin Wang. 2023. Lea2: A lightweight ensemble adversarial attack via non-overlapping vulnerable frequency regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4510–4521. find
  33. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  34. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
  35. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  36. Haodi Wang, Kai Dong, Zhilei Zhu, Haotong Qin, Aishan Liu, Xiaolin Fang, Jiakai Wang, and Xianglong Liu. 2024a. Transferable multimodal attack on vision-language pre-training models. In 2024 IEEE Symposium on Security and Privacy (SP), pages 1722–1740. IEEE. find
  37. Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang, Lingyi Hong, Pinxue Guo, Haijing Guo, and Wenqiang Zhang. 2024b. Boosting the transferability of adversarial attacks with global momentum initialization. Expert Systems with Applications, 255:124757. find
  38. Xiaosen Wang and Kun He. 2021. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1924–1933. find
  39. Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. 2021. Admix: Enhancing the transferability of adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16158–16167. find
  40. Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and Yu-Wing Tai. 2020. Boosting the transferability of adversarial samples via attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1161–1170. find
  41. Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. 2022. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14983–14992. find
  42. Fan Yang, Yihao Huang, Kailong Wang, Ling Shi, Geguang Pu, Yang Liu, and Haoyu Wang. 2024. Efficient and effective universal adversarial attack against vision-language pre-training models. arXiv preprint arXiv:2410.11639. — [arXiv:2410.11639](https://arxiv.org/abs/2410.11639) find
  43. Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. 2022a. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671–15680. adver… find
  44. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer.
  45. Yan Zeng, Xinsong Zhang, and Hang Li. 2021. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276. — [arXiv:2111.08276](https://arxiv.org/abs/2111.08276) find
  46. Chaoning Zhang, Philipp Benz, Adil Karjauv, and In So Kweon. 2021. Data-free universal adversarial perturbation and black-box attack. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7868–7877. find
  47. Jiaming Zhang, Qi Yi, and Jitao Sang. 2022a. Towards adversarial attack on vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5005–5013. find
  48. Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. 2022b. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14993–15002. find
  49. Peng-Fei Zhang, Zi Huang, and Guangdong Bai. 2024. Universal adversarial perturbations for vision-language pre-trained models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 862–871. find
  50. Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. 2022c. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning, pages 26693–26712. PMLR. find
  51. Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, and Hai Jin. 2023. Advclip: Downstream-agnostic adversarial examples in multimodal contrastive learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6311–6320. find
  52. Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. 2022. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804– 1… find
  53. Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. 2022b. Boosting transferability of targeted find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-asymmetric-relational-geometry-driven-universal/