Elamur Research⌕
☰
home / papers / 2026 / SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models

SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models

Paper · 2026 · Xinyi Zeng, Xue Yang, Jingyuan Zhang et al.

Summary

Multimodal LLMs are easier to jailbreak than text-only models, especially when the harmful content is hidden in an image. Existing defenses either need expensive fine-tuning with annotated data or rewrite inputs and outputs, which adds compute and can hurt helpfulness. The authors first check whether MLLMs (Qwen2.5-VL, Qwen3-VL) internally separate harmful from benign queries. PCA/t-SNE plots of hidden states show a clear split at prefill and during the first few decoding steps. Image-borne attacks sit closer to the benign cluster, which suggests weaker cross-modal safety alignment.

Building on this, SafeSteer adds two components at decoding time:

  • Decoding-Probe: a logistic-regression probe on PCA-projected hidden states. It scores candidate tokens for harmfulness and reranks them at each step, so generation is pushed toward safe continuations.
  • Modal Semantic Alignment Vector: derived from the representational gap between image-based and text-based attacks, and injected after prefill. It carries text-side safety over to images.

Without fine-tuning, SafeSteer reportedly raises safety by up to 33.4% while preserving utility, with little added overhead.

Key points

  • Analysis dataset: 200 benign MM-Vet samples plus 200 harmful ones. The harmful set has 50 each of SD, TYPO and SDTYPO from MM-SafetyBench "01-Illegal Activity", and 50 text-only "CB" prompts paired with blank images.
  • Evaluation metrics:
    • Safety: ASR judged by LlamaGuard-3.
    • Utility: MM-Vet accuracy scored by GPT-4, and refusal rate on VL-Guard Safe.
    • Efficiency: training and inference time.
  • Baselines: ECSO, MLLM-Protector and MRD. SafeSteer also generalizes to out-of-domain FigStep and VL-Guard.
  • Training cost: only a linear probe is fitted, with the MLLM frozen; experiments ran on a single NVIDIA A800.
  • Ablations: the Decoding-Probe mainly helps against text-based (CB) attacks, while the alignment vector is key for image-based (SD) attacks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓Beihang UniversityBeijing · China86 works · 9,097 citations🎓Wuhan UniversityWuhan · China47 works · 1,246 citations🏢KuaishouBeijing · China19 works · 95 citations🎓Chongqing UniversityChongqing · China14 works · 318 citations🎓Nanjing University of Aeronautics and AstronauticsNanjing · China6 works · 2 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan, Xiang Chen, Kaiwen Wei, Hankun Kang and Yu Tian. 2026. SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.916

@inproceedings{zeng2026safesteer,
  title = {{SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models}},
  author = {Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan, Xiang Chen, Kaiwen Wei, Hankun Kang and Yu Tian},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.916},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.916}
}

References

2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-safesteer-decoding-level-defense-mechanism/