Elamur Research⌕
☰
home / papers / 2026 / Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Paper · 2026 · Qishun Yang, Shu Yang, Lijie Hu et al.

Summary

Multimodal LLMs often lose the safety behaviour of their text-only backbones, because harmful images can carry or amplify malicious intent past textual safeguards. Existing defenses rely on safety-labelled or contrastive data. Such data is hard to build for vision: dangers are easy to depict, but abstract notions like harmlessness have no visual counterpart. The authors propose Visual Self-Fulfilling Alignment (VSFA), which reverses the logic of emergent misalignment. They fine-tune VLMs only on ordinary, safety-neutral VQA pairs (for example, describing what appears in a picture) built around synthetic threat-themed images. The images come from arXiv AI-safety abstracts that were turned into image prompts. The idea is that repeated exposure to threatening scenes nudges the model toward a cautious, vigilant persona without any explicit refusal supervision. Across four VLMs and several jailbreak benchmarks, VSFA lowers attack success rate and yields more constructive answers than the AdaShield and VLGuard baselines. It also refuses benign requests far less often and keeps general skills nearly intact. The work points to a label-free, cheap route to multimodal alignment.

Key points

  • Training data pipeline: arXiv AI-safety abstracts are converted into image prompts by GPT-4o-mini. The Doubao API then renders 700 synthetic threat images, which are paired with neutral VQA questions.
  • Models and judging: Qwen3-VL 8B, Qwen2.5-VL-7B, LLaVA-v1.6-7B and LLaVA-1.5-7B were tested on three jailbreak-oriented safety benchmarks, with GPT-4o as judge.
  • Over-refusal and general capability: On MM-Vet, VLGuard scores slightly higher on capability, but VSFA's benign-query refusal rate is about 3–4× lower. The drop on MMLU and MMMU stays under 0.5%.
  • Ablations: Image-only training outperforms text-only and mixed training, beating Mixed by 4–7% ASR on every model. The safety effect holds across 12 visual styles.
  • Mechanistic analysis: An SAE analysis finds a "safety-oriented persona" latent. Steering confirms its causal role in both directions: adding it lowers ASR, and removing it raises ASR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia30 works · 1,270 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Qishun Yang, Shu Yang, Lijie Hu and Di Wang. 2026. Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.490

@inproceedings{yang2026visual,
  title = {{Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images}},
  author = {Qishun Yang, Shu Yang, Lijie Hu and Di Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.490},
  url = {https://doi.org/10.18653/v1/2026.acl-long.490}
}

References

80 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andy Arditi and Runjin Chen. 2025. Finding “misaligned persona” features in open-weight models. LessWrong. Published September 9, 2025. find
  2. Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. 2024. Image hijacks: Adversarial images can control generative models at runtime. pages 2443– 2455. find
  3. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. 2025. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. find
  4. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/ paper/index.html. ↗
  5. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… find
  6. Rajat Chawla, Arkajit Datta, Tushar Verma, Adarsh Jha, Anmol Gautam, Ayush Vatsal, Sukrit Chaterjee, Mukunda NS, and Ishaan Bhola. 2024. Veagle: Advancements in multimodal representation learning. CoRR, abs/2403.08773. — [arXiv:2403.08773](https://arxiv.org/abs/2403.08773) find
  7. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684.
  8. Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, pages 23951–23959. AAAI Pre… find
  9. Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu, Min Yang, Shujian Huang, Lidia S Chao, and Derek F Wong. 2026. Neuron-aware data selection in instruction tuning for large language models. arXiv preprint arXiv:2603.13201. — [arXiv:2603.13201](https://arxiv.org/abs/2603.13201) find
  10. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, volume 2024, pages 32694–32717. find
  11. Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, and Sijia Liu. 2025. Safety mirage: How spurious correlations undermine vlm safety fine-tuning and can be mitigated by machine unlearning. find
  12. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. CoRR, arXiv:2410.20526.
  13. Ruoxi Cheng, Yizhong Ding, Shuirong Cao, Ranjie Duan, Xiaoshuang Jia, Shaowei Yuan, Simeng Qin, Zhiqiang Wang, and Xiaojun Jia. 2025. Pbi-attack: Prior-guided bimodal interactive black-box jailbreak attack for toxicity maximization. pages 609–628. find
  14. Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, and Owain Evans. 2025. Subliminal learning: Language models transmit behavioral traits via hidden signals in data. CoRR, abs/2507.14805. — [arXiv:2507.14805](https://arxiv.org/abs/2507.14805) find
  15. Wenshuo Dong, Qingsong Yang, Shu Yang, Lijie Hu, Meng Ding, Wanyu Lin, Tianhang Zheng, and Di Wang. 2025. Understanding and mitigating cross-lingual privacy leakage via language-specific and universal privacy neurons. CoRR, abs/2506.00759. — [arXiv:2506.00759](https://arxiv.org/abs/2506.00759) find
  16. Ranjie Duan, Jiexi Liu, Xiaojun Jia, Shiji Zhao, Ruoxi Cheng, Fengxiang Wang, Cheng Wei, Yong Xie, Chang Liu, Defeng Li, Yinpeng Dong, Yichi Zhang, Yuefeng Chen, Chongwen Wang, Xingjun Ma, Xingxing Wei, Yang Liu, Hang Su, Jun Zhu, and 11 others. 2025. Oyster-i: Beyond refusal - constructive safety a… — [arXiv:2509.01909](https://arxiv.org/abs/2509.01909) find
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  18. Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, and Yaoqing Yang. 2025. Why LLM safety guardrails collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets. CoRR, abs/2506.05346.
  19. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  20. Jingyu Hu, Shu Yang, Xilin Gong, Hongming Wang, Weiru Liu, and Di Wang. 2025a. MONICA: real-time monitoring and calibration of chain-of-thought sycophancy in large reasoning models. CoRR, abs/2511.06419. — [arXiv:2511.06419](https://arxiv.org/abs/2511.06419) find
  21. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. find
  22. Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang. 2024. A hopfieldian view-based interpretation for chain-of-thought reasoning. CoRR, abs/2406.12255. — [arXiv:2406.12255](https://arxiv.org/abs/2406.12255) find
  23. Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, and Lucas Dixon. 2024. Who’s asking? user personas and the mechanics of latent misalignment. find
  24. Xuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang, and Jing Shao. 2025b. Vlsbench: Unveiling visual leakage in multimodal safety. pages 8285–8316.
  25. Jeremiah Giordani. 2025. Re-emergent misalignment: How narrow fine-tuning erodes safety alignment in llms. CoRR, abs/2507.03662. — [arXiv:2507.03662](https://arxiv.org/abs/2507.03662) find
  26. Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. 2025. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. find
  27. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2025. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Xiaojun Jia, Jie Liao, Qi Guo, Teng Ma, Simeng Qin, Ranjie Duan, Tianlin Li, Yihao Huang, Zhitao Zeng…
  28. Xinke Jiang, Yue Fang*, Rihong Qiu*, Haoyu Zhang, Yongxin Xu, Hao Chen, Wentao Zhang, Ruizhe Zhang, Yuchen Fang, Xu Chu, and 1 others. 2025a. Tc-rag: Turing-complete rag’s case study on medical llm systems.
  29. Xinke Jiang, Ruizhe Zhang*, Yongxin Xu*, Rihong Qiu*, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, and 1 others. 2025b. Hykge: A hypothesis knowledge graph enhanced framework for accurate and reliable medical llms responses. find
  30. Xinyan Jiang, Wenjing Yu, Di Wang, and Lijie Hu. 2026. Global evolutionary steering: Refining activation steering control via cross-layer consistency. arXiv preprint arXiv:2603.12298. — [arXiv:2603.12298](https://arxiv.org/abs/2603.12298) find
  31. Xinyan Jiang, Lin Zhang, Jiayi Zhang, Qingsong Yang, Guimin Hu, Di Wang, and Lijie Hu. 2025c. MSRS: adaptive multi-subspace representation steering for attribute alignment in large language models. CoRR, abs/2508.10599. — [arXiv:2508.10599](https://arxiv.org/abs/2508.10599) find
  32. Yilei Jiang, Yingshui Tan, and Xiangyu Yue. 2024. Rap-guard: Safeguarding multimodal large language models via rationale-aware defensive prompting. CoRR, abs/2412.18826. — [arXiv:2412.18826](https://arxiv.org/abs/2412.18826) find
  33. Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L. Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025. Programming refusal with conditional activation steering. find
  34. Hongji Li, Junchi Yao, Manjiang Yu, Priyanka Singh, Xue Li, Di Wang, and Lijie Hu. 2025a. Towards reasoning-preserving unlearning in multimodal large language models. CoRR, abs/2512.17911. — [arXiv:2512.17911](https://arxiv.org/abs/2512.17911) find
  35. Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025b. Safety layers in aligned large language models: The key to LLM security. find
  36. Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. pages 174–189. find
  37. Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, N… find
  38. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. find
  39. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning.
  40. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning.
  41. Qin Liu, Fei Wang, Chaowei Xiao, and Muhao Chen. 2025. Vlm-guard: Safeguarding vision-language models via fulfilling safety alignment gap. CoRR, abs/2502.10486. — [arXiv:2502.10486](https://arxiv.org/abs/2502.10486) find
  42. Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024b. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LVI, Lecture Not…
  43. Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024c. Safety of multimodal large language models on images and text. pages 8151–8159. Survey Track. find
  44. Robert K. Merton. 1948. The self-fulfilling prophecy. The Antioch Review, 8(2):193–210. nostalgebraist. 2020. interpreting GPT: the logit lens. LessWrong. find
  45. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  46. CoRR, Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge J. Belongie, and Zeynep Akata. 2025. Sparse autoencoders learn monosemantic features in vision-language models. CoRR, abs/2504.02821. — [arXiv:2504.02821](https://arxiv.org/abs/2504.02821) find
  47. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with GPT-4. CoRR, abs/2304.03277.
  48. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2024. Visual adversarial examples jailbreak aligned large language models. pages 21527–21536.
  49. Qwen. 2025. Qwen3-vl technical report. abs/2511.21631.
  50. CoRR, Kaixuan Ren, Preslav Nakov, and Usman Naseem. 2025. Dual-bench: Measuring over-refusal and robustness in vision-language models. CoRR, abs/2510.10846. — [arXiv:2510.10846](https://arxiv.org/abs/2510.10846) find
  51. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. pages 10674–10685.
  52. Peiyuan Tang, Haojie Xin, Xiaodong Zhang, Jun Sun, Qin Xia, and Zijiang Yang. 2025. The safety reminder: A soft prompt to reactivate delayed safety awareness in vision-language models. CoRR, abs/2506.15734. — [arXiv:2506.15734](https://arxiv.org/abs/2506.15734) find
  53. Gemma Team. 2025. Gemma 3 technical report. CoRR, arXiv:2503.19786. find
  54. Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
  55. Alex Turner. 2025. Self-fulfilling misalignment: Data might be poisoning our ai models. Blog post. find
  56. Keyu Wang, Jin Li, Shu Yang, Zhuoran Zhang, and Di Wang. 2026. When truth is overridden: Uncovering the internal origins of sycophancy in large language models. pages 33566–33574. find
  57. Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. 2025. Persona features control emergent misalignment. CoRR, abs/2506.19823. — [arXiv:2506.19823](https://arxiv.org/abs/2506.19823) find
  58. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the…
  59. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  60. Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. 2024b. Adashield : Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings,… find
  61. Siqi Wen, Shu Yang, Shaopeng Fu, Jingfeng Zhang, Lijie Hu, and Di Wang. 2026. Concept-based dictionary learning for inference-time safety in vision language action models. arXiv preprint arXiv:2602.01834. — [arXiv:2602.01834](https://arxiv.org/abs/2602.01834) find
  62. Haodong Xie, Rahul Singh Maharjan, Federico Tavella, and Angelo Cangelosi. 2024. From concrete to abstract: A multimodal generative approach to abstract concept learning. volume abs/2410.02365. — [arXiv:2410.02365](https://arxiv.org/abs/2410.02365) find
  63. Haochuan Xu, Yun Sing Koh, Shuhuai Huang, Zirun Zhou, Di Wang, Jun Sakuma, and Jingfeng Zhang. 2025. Model-agnostic adversarial attack and defense for vision-language-action models. arXiv preprint arXiv:2510.13237. — [arXiv:2510.13237](https://arxiv.org/abs/2510.13237) find
  64. Shu Yang, Jiayuan Su, Han Jiang, Mengdi Li, Keyuan Cheng, Muhammad Asif Ali, Lijie Hu, and Di Wang. 2024a. Dialectical alignment: Resolving the tension of 3h and security threats of llms. CoRR, abs/2404.00486. — [arXiv:2404.00486](https://arxiv.org/abs/2404.00486) find
  65. Shu Yang, Shenzhe Zhu, Liang Liu, Lijie Hu, Mengdi Li, and Di Wang. 2024b. Exploring the personality traits of llms through latent features steering. arXiv preprint arXiv:2410.10863. — [arXiv:2410.10863](https://arxiv.org/abs/2410.10863) find
  66. Tiancheng Yang, Lin Zhang, Jiaye Lin, Guimin Hu, Di Wang, and Lijie Hu. 2025. D-LEAF: localizing and correcting hallucinations in multimodal llms via layer-to-head attention diagnostics. CoRR, abs/2509.07864. — [arXiv:2509.07864](https://arxiv.org/abs/2509.07864) find
  67. Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. 2025. Understanding the repeat curse in large language models from a feature perspective. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7787–7815. find
  68. Manjiang Yu, Hongji Li, Priyanka Singh, Xue Li, Di Wang, and Lijie Hu. 2025. PIXEL: adaptive steering via position-wise injection with exact estimated levels under subspace calibration. CoRR, abs/2510.10205. — [arXiv:2510.10205](https://arxiv.org/abs/2510.10205) find
  69. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. pages 57730–57754.
  70. Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. MMMU: A massive multi-discipline multimodal u…
  71. Jiahao Zhang, Zeqing Zhang, Di Wang, and Lijie Hu. 2026a. Controlling repetition in protein language models. arXiv preprint arXiv:2602.00782. — [arXiv:2602.00782](https://arxiv.org/abs/2602.00782) find
  72. Qi Zhang, Yifei Wang, Jingyi Cui, Xiang Pan, Qi Lei, Stefanie Jegelka, and Yisen Wang. 2025a. Beyond interpretability: The gains of feature monosemanticity on model robustness. find
  73. Ruizhe Zhang, Xinke Jiang, Zhibang Yang, Zhixin Zhang, Jiaran Gao, Yuzhen Xiao, Hongbin Lai, Xu Chu, Junfeng Zhao, and Yasha Wang. 2026b. Stackplanner: A centralized hierarchical multi-agent system with task-experience memory management. find
  74. Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, and Jing Shao. 2025b. SPA-VL: A comprehensive safety preference alignment dataset for vision language models. pages 19867–19878. find
  75. Yuyou Zhang, Miao Li, William Han, Yihang Yao, Zhepeng Cen, and Ding Zhao. 2025c. Safety is not only about refusal: Reasoning-enhanced fine-tuning for interpretable LLM safety. pages 18727–18746. find
  76. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. Lmsys-chat-1m: A large-scale real-world LLM conversation dataset.
  77. Wenrui Zhou, Shu Yang, Qingsong Yang, Zikun Guo, Lijie Hu, and Di Wang. 2025a. Flattery in motion: Benchmarking and analyzing sycophancy in video-llms. CoRR, abs/2506.07180.
  78. Zirun Zhou, Zhengyang Xiao, Haochuan Xu, Jing Sun, Di Wang, and Jingfeng Zhang. 2025b. Goal-oriented backdoor attack against vision-language-action models via physical objects. arXiv preprint arXiv:2510.09269. — [arXiv:2510.09269](https://arxiv.org/abs/2510.09269) find
  79. Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy M. Hospedales. 2024. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. pages 62867–62891.
  80. Xiaohan Zou, Jian Kang, George Kesidis, and Lu Lin. 2025. Understanding and rectifying safety perception distortion in vlms. CoRR, abs/2502.13095. — [arXiv:2502.13095](https://arxiv.org/abs/2502.13095) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-visual-self-fulfilling-alignment-shaping/