Elamur Research⌕
☰
home / papers / 2026 / B-APO: Bias-Targeted Adversarial Preference Optimization for Debiasing Multimodal Large Language Models

B-APO: Bias-Targeted Adversarial Preference Optimization for Debiasing Multimodal Large Language Models

Paper · 2026 · Pinlong Zhao, Zike Ding, Zengshu Ye et al.

Summary

Multimodal LLMs often lean on one modality and ignore the other. Language bias means answering from textual priors instead of the image. Vision bias means describing salient visual content that does not address the question. Earlier preference-based debiasing, notably NaPO, builds negative examples by masking out a whole modality. That gives an all-or-nothing training signal that differs from how bias shows up in practice, where both inputs exist but are noisy or ambiguous.

B-APO treats debiasing as a min-max game. In the attack stage, small FGSM/PGD-style perturbations are applied in latent space:

  • Vision side: connector-output visual tokens are perturbed toward a language-only prior.
  • Text side: question-token embeddings are perturbed toward a vision-only prior.

A bias filter checks that each resulting hard negative actually shows the intended bias. In the defense stage, preference optimization widens the margin between clean and adversarial responses.

Across bias and hallucination benchmarks, B-APO outperforms NaPO, keeps general skills, and holds up better under corrupted inputs. This suggests targeted adversarial negatives are a more realistic debiasing signal than masking.

Key points

  • Benchmarks: evaluated on VLind-Bench, Object HalBench, AMBER and MMHalBench. B-APO gains +1.8% on Language Prior and +1.8% on Commonsense Bias over NaPO. Further checks on InternVL-Chat-V1.5-7B and NoCaps captioning indicate it transfers across architectures and tasks.
  • Attack effectiveness:
    • Vision-side attacks raise the Language Bias Score from 12% to 58%.
    • Text-side attacks raise the Vision Bias Score from 8% to 52%.
    • Cross-modal grounding stays moderate (42% and 38%), whereas masking drives it to roughly zero.
  • Perturbation budget: ϵ = 0.05 gives the best trade-off. Values above about 0.07 hurt debiasing, likely because training becomes unstable.
  • Robustness: on 500 corrupted VLind-Bench samples (Gaussian noise, JPEG compression, domain-shifted text), B-APO degrades less than NaPO. Corrupted-set scores as read from Table 4 are approximately CB 45.8 vs 44.0 and LP 41.2 vs 38.5; the extracted table was garbled, so exact column assignment is uncertain.
  • Cost and limits:
    • Training takes about 1.3× as long, about 1.2× with perturbation caching, and about 1.12× in an FGSM-only setup; inference cost is unchanged.
    • Gains are weak on fine-grained tasks such as counting (+1.7%).
    • Evaluation is English-only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hangzhou Dianzi UniversityHangzhou · China15 works · 1,061 citations🎓Goldsmiths University of LondonLondon · United Kingdom5 works · 381 citations

Also on the byline: Zhejiang Guangsha Vocational and Technical University of Construction.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Pinlong Zhao, Zike Ding, Zengshu Ye and Zhou Zhaoting. 2026. B-APO: Bias-Targeted Adversarial Preference Optimization for Debiasing Multimodal Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1843

@inproceedings{zhao2026b,
  title = {{B-APO: Bias-Targeted Adversarial Preference Optimization for Debiasing Multimodal Large Language Models}},
  author = {Pinlong Zhao, Zike Ding, Zengshu Ye and Zhou Zhaoting},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1843},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1843}
}

References

34 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971–4980. find
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.
  3. Remi Cadene, Corentin Dancette, Hedi Ben younes, Matthieu Cord, and Devi Parikh. 2019. Rubi: Reducing unimodal biases for visual question answering. In Advances in neural information processing systems, volume 32. find
  4. Meiqi Chen, Yixin Cao, Yan Zhang, and Chaochao Lu. 2024a. Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16449– 16469. find
  5. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. pages 24185–24198.
  6. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. volume 36, pages 49250–49267.
  7. Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. Mutant: A training paradigm for out-of-distribution generalization in visual question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 878–892. find
  8. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  9. Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. 2022. Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5078–5088. find
  10. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis…
  11. Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. Vlind-bench: Measuring language priors in large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4129–4144. find
  12. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742.
  13. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  14. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
  15. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725.
  16. Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12700– 12710. find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  18. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  19. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045.
  20. Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Arık, and Tomas Pfister. 2024. find
  21. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654. — [arXiv:2405.18654](https://arxiv.org/abs/2405.18654) find
  22. Christian Schlarmann and Matthias Hein. 2023. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677– 3685. find
  23. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2024. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–…
  24. Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mdpo: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8078–8088.
  25. Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. 2023. Amber:an llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397. optimization. In Proceedings of the IEEE/CVF Conference on…
  26. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) find
  27. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations.
  28. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
  29. Xi Zhu, Zhendong Mao, Chunxiao Liu, Peng Zhang, Bin Wang, and Yongdong Zhang. 2021. Overcoming language priors with self-supervised learning for visual question answering. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages… find
  30. Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2025. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pag… find
  31. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, and 1 others. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.
  32. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and 1 others. 2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vis…
  33. Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, and 1 others. 2025. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19985–19995. find
  34. Zefeng Zhang, Hengzhu Tang, Jiawei Sheng, Zhenyu Zhang, Yiming Ren, Zhenyang Li, Dawei Yin, Duohe Ma, and Tingwen Liu. 2025. Debiasing multimodal large language models via noise-aware preference find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-b-apo-bias-targeted-adversarial/