Elamur Research⌕
☰
home / papers / 2026 / TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs

TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs

Paper · 2026 · Xiaofeng Zhang, Yuanchao Zhu, Qiyan Zhao et al.

Summary

Large vision-language models (LVLMs) often mention objects that are not in the image. This undermines trust in uses such as captioning and product descriptions. The authors link part of this to two effects. First, attention collapses onto a few visual "sink" tokens with extreme query/key activations. Second, RoPE makes attention to early image tokens fade as the generated text grows longer. They show that removing or flattening sink attention, as Vissink does, helps short-answer QA but hurts long captioning, because these high-activation tokens carry context the model needs. Their decoding-time method, called TokenTruth in the body (TokenPenalty in the title), works at two levels. A dynamic ALiBi-style penalty inside the causal mask redistributes excess attention and counters long-range decay. A head-level enhancement then rebalances attention without replacing heads. Tested on LLaVA-1.5 and Qwen-family models, it reduces hallucination and improves perception and VQA results. The work suggests that sinks should be rebalanced rather than eliminated.

Key points

  • Diagnostic finding: Vissink-style intervention on maximum-activation tokens raises GQA and SQA scores but lowers CHAIR results. The authors read this as evidence that sink tokens are needed for contextual understanding.
  • Positional decay: LLaVA1.5 and Qwen2-VL lose attention to visual information during long generation. The paper relates this to CCA-LLaVA, MCA-LLaVA and Farsight.
  • How it differs from prior work:
    • Farsight uses static causal masks, whereas TokenTruth adjusts the mask dynamically.
    • EAH replaces attention heads, whereas TokenTruth redistributes attention within them.
    • Vissink targets BOS tokens, whereas TokenTruth works on image tokens and individual heads.
  • Evaluation: The method was tested on LLaVA-1.5-7B, Qwen2-VL-7B and Qwen2.5-VL-7B, using MME, several VQA benchmarks and CHAIR. The authors report gains on all benchmarks over the LLaVA-1.5 baseline and measurable gains for Qwen2-VL, especially on VQA.
  • Ablation and limitations:
    • Of the global decay values tested (σ ∈ {0.7, 0.8, 0.9}), 0.8–0.9 gave the best trade-off.
    • The authors note that activation-based token selection may miss semantic importance.
    • Multi-image and video inputs are left untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓Jilin UniversityChangchun · China30 works · 5,252 citations

Also on the byline: Huizhou University.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Xiaofeng Zhang, Yuanchao Zhu, Qiyan Zhao, Xiaosong Yuan, Jiawei Cao and Xuhang Chen. 2026. TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.628

@inproceedings{zhang2026tokenpenalty,
  title = {{TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs}},
  author = {Xiaofeng Zhang, Yuanchao Zhu, Qiyan Zhao, Xiaosong Yuan, Jiawei Cao and Xuhang Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.628},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.628}
}

References

103 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. 2024. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718. — [arXiv:2406.12718](https://arxiv.org/abs/2406.12718) find
  3. Kyungho Bae, Jinhyung Kim, Sihaeng Lee, Soonyoung Lee, Gunhee Lee, and Jinwoo Choi. 2025. Mash-vlm: Mitigating action-scene hallucination in video-llms through disentangled spatial-temporal representations. arXiv preprint arXiv:2503.15871. — [arXiv:2503.15871](https://arxiv.org/abs/2503.15871) find
  4. Jiaqi Bai, Hongcheng Guo, Zhongyuan Peng, Jian Yang, Zhoujun Li, Mohan Li, and Zhihong Tian. 2025. Mitigating hallucinations in large vision-language models by adaptively constraining information flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23442–23450. find
  5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.
  6. Jinhe Bi, Yifan Wang, Danqi Yan, Aniri, Wenke Huang, Zengjie Jin, Xiaowen Ma, Artur Hecker, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. 2025a. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. Preprint, arXiv:2502.12119. — [arXiv:2502.12119](https://arxiv.org/abs/2502.12119) find
  7. Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025b. LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti…
  8. Liwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin, Ruixiang Tang, and Vladimir Pavlovic. 2025. Eazy: Eliminating hallucinations in lvlms by zeroing out hallucinatory image tokens. arXiv preprint arXiv:2503.07772. — [arXiv:2503.07772](https://arxiv.org/abs/2503.07772) find
  9. Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. 2024a. Alleviating hallucinations in large vision-language models through hallucination-induced optimization. arXiv preprint arXiv:2405.15356. — [arXiv:2405.15356](https://arxiv.org/abs/2405.15356) find
  10. Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. 2025a. Attention hijackers: Detect and disentangle attention hijacking in lvlms for hallucination mitigation. arXiv preprint arXiv:2503.08216. — [arXiv:2503.08216](https://arxiv.org/abs/2503.08216) find
  11. Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025b. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486. — [arXiv:2503.06486](https://arxiv.org/abs/2503.06486) find
  12. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024b. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. 18th European Conference on Computer Vision ECCV 2024.
  13. Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024c. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425. — [arXiv:2403.00425](https://arxiv.org/abs/2403.00425) find
  14. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883.
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  16. Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. 2025. Truth-print: Mitigating lvlm object hallucination via latent truthful-guided pre-intervention. arXiv preprint arXiv:2503.10602. — [arXiv:2503.10602](https://arxiv.org/abs/2503.10602) find
  17. Peng Gao, Yujian Lee, Xiaofeng Zhang, Zailong Chen, and Hui Zhang. 2025. Remember me: Bridging the long-range gap in lvlms with three-step inference-only decay resilience strategies. arXiv preprint arXiv:2511.09868. — [arXiv:2511.09868](https://arxiv.org/abs/2511.09868) find
  18. Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024a. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514. find
  19. Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024b. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514. find
  20. Jihao Gu, Yingyao Wang, Meng Cao, Pi Bu, Jun Song, Yancheng He, Shilong Li, and Bo Zheng. 2024. Token preference optimization with self-calibrated visual-anchored rewards for hallucination mitigation. arXiv preprint arXiv:2412.14487. find
  21. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617. find
  22. Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, and Jinqiao Wang. 2024. Cracking the code of hallucination in lvlms with vision-aware head divergence. arXiv preprint arXiv:2412.13949.
  23. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.
  24. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis…
  25. Qing Li. 2025a. Kore: Enhancing knowledge injection for large multimodal models via knowledge-oriented augmentations and constraints. Preprint, arXiv:2510.19316. — [arXiv:2510.19316](https://arxiv.org/abs/2510.19316) find
  26. Kailin Jiang, Ning Jiang, Yuntao Du, Yuchen Ren, Yuchen Li, Yifan Gao, Jinhe Bi, Yunpu Ma, Qingqing Liu, Xianhao Wang, Yifan Jia, Hongbo Jiang, Yao-cong Hu, Bin Li, and Lei Liu. 2025b. Mined: Probing and updating with multimodal time-sensitive knowledge for large multimodal models. Preprint, arXiv:2… find
  27. Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2024b. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724. — [arXiv:2411.16724](https://arxiv.org/abs/2411.16724) find
  28. Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. 2025. Massive values in self-attention modules are the key to contextual knowledge understanding. arXiv preprint arXiv:2502.01563. — [arXiv:2502.01563](https://arxiv.org/abs/2502.01563) find
  29. Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, and Qingmin Liao. 2024. Catch: Complementary adaptive token-level contrastive decoding to mitigate hallucinations in lvlms. arXiv preprint arXiv:2411.12713. — [arXiv:2411.12713](https://arxiv.org/abs/2411.12713) find
  30. Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321. — [arXiv:2503.03321](https://arxiv.org/abs/2503.03321) find
  31. Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. 2024. Code: Contrasting self-generated description to combat hallucination in large multi-modal models. arXiv preprint arXiv:2406.01920. — [arXiv:2406.01920](https://arxiv.org/abs/2406.01920) find
  32. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13…
  33. Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. 2025. Self-introspective decoding: Alleviating hallucinations for large vision-language models. In The Thirteenth International Conference on Learning Representations. find
  34. Bin Li, Dehong Gao, Yeyuan Wang, Linbo Jin, Shanqing Yu, Xiaoyan Cai, and Libin Yang. 2025a. Instruction-aligned visual attention for mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2503.18556. — [arXiv:2503.18556](https://arxiv.org/abs/2503.18556) find
  35. Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024a. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa… find
  36. Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. 2025b. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding. arXiv preprint arXiv:2501.01926. — [arXiv:2501.01926](https://arxiv.org/abs/2501.01926) find
  37. Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, and Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, and Yue Zhao. 2025c. Treble counterfactual vlms: A causal approach to hallucination. arXiv preprint arXiv:2503.06169. find
  38. Shuo Li, Jiajun Sun, Guodong Zheng, Xiaoran Fan, Yujiong Shen, Yi Lu, Zhiheng Xi, Yuming Yang, Wenming Tan, Tao Ji, et al. 2025d. Mitigating object hallucinations in mllms via multi-frequency perturbations. arXiv preprint arXiv:2503.14895. find
  39. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.
  40. Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. 2025e. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03628. — [arXiv:2502.03628](https://arxiv.org/abs/2502.03628) find
  41. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024a. Visual instruction tuning. Advances in neural information processing systems, 36.
  42. Shi Liu, Kecheng Zheng, and Wei Chen. 2024b. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. arXiv preprint arXiv:2407.21771. — [arXiv:2407.21771](https://arxiv.org/abs/2407.21771) find
  43. Shi Liu, Kecheng Zheng, and Wei Chen. 2024c. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125–140. Springer. find
  44. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  45. Maria Lymperaiou, Giorgos FIlandrianos, Angeliki Dimitriou, Athanasios Voulodimos, and Giorgos Stamou. 2025. Halcece: A framework for explainable hallucination detection through conceptual counterfactuals in image captioning. arXiv preprint arXiv:2503.00436. — [arXiv:2503.00436](https://arxiv.org/abs/2503.00436) find
  46. Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. 2024. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13151–13160. find
  47. Shunqi Mao, Chaoyi Zhang, and Weidong Cai. 2025. Through the magnifying glass: Adaptive perception magnification for hallucination-free vlm decoding. arXiv preprint arXiv:2503.10183.
  48. Dexter Neo and Tsuhan Chen. 2024. Vord: Visual ordinal calibration for mitigating object hallucinations in large vision-language models. arXiv preprint arXiv:2412.15739. — [arXiv:2412.15739](https://arxiv.org/abs/2412.15739) find
  49. Tianfan Peng, Yuntao Du, Pengzhou Ji, Shijie Dong, Kailin Jiang, Mingchuan Ma, Yijun Tian, Jinhe Bi, Qian Li, Wei Du, Feng Xiao, and Lizhen Cui. 2025. Can visual input be compressed? a visual token compression benchmark for large multimodal models. Preprint, arXiv:2511.02650. — [arXiv:2511.02650](https://arxiv.org/abs/2511.02650) find
  50. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156.
  51. Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, and Mang Ye. 2025. Backdoor cleaning without external guidance in mllm fine-tuning. arXiv preprint arXiv:2505.16916. — [arXiv:2505.16916](https://arxiv.org/abs/2505.16916) find
  52. Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Arık, and Tomas Pfister. 2024. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654. — [arXiv:2405.18654](https://arxiv.org/abs/2405.18654) find
  53. Yuying Shang, Xinyi Zeng, Yutao Zhu, Xiao Yang, Zhengwei Fang, Jingyuan Zhang, Jiawei Chen, Zinan Liu, and Yu Tian. 2024. From pixels to tokens: Revisiting object hallucinations in large vision-language models. arXiv preprint arXiv:2410.06795. — [arXiv:2410.06795](https://arxiv.org/abs/2410.06795) find
  54. Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762.
  55. Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang. 2025. Octopus: Alleviating hallucination via dynamic contrastive decoding. arXiv preprint arXiv:2503.00361. — [arXiv:2503.00361](https://arxiv.org/abs/2503.00361) find
  56. Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. 2025a. Intervening anchor token: Decoding strategy in alleviating hallucinations for MLLMs. In The Thirteenth International Conference on Learning Representations. find
  57. Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. 2025b. Seeing far and clearly: Mitigating hallucinations in mllms with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition Confere… find
  58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  59. Chongjun Tu, Peng Ye, Dongzhan Zhou, Lei Bai, Gang Yu, Tao Chen, and Wanli Ouyang. 2025. Attention reallocation: Towards zero-cost and controllable hallucination mitigation of mllms. arXiv preprint arXiv:2503.08342. — [arXiv:2503.08342](https://arxiv.org/abs/2503.08342) find
  60. Zifu Wan, Ce Zhang, Silong Yong, Martin Q Ma, Simon Stepputtis, Louis-Philippe Morency, Deva Ramanan, Katia Sycara, and Yaqi Xie. 2025. Only: One-layer intervention sufficiently mitigates hallucinations in large vision-language models. arXiv preprint arXiv:2507.00898. — [arXiv:2507.00898](https://arxiv.org/abs/2507.00898) find
  61. Chao Wang, Weiwei Fu, and Yang Zhou. 2025a. Tpc: Cross-temporal prediction connection for vision-language model hallucination reduction. arXiv preprint arXiv:2503.04457. — [arXiv:2503.04457](https://arxiv.org/abs/2503.04457) find
  62. Jiaqi Wang, Yifei Gao, and Jitao Sang. 2024a. Valid: Mitigating the hallucination of large vision language models by visual layer fusion contrastive decoding. arXiv preprint arXiv:2411.15839. — [arXiv:2411.15839](https://arxiv.org/abs/2411.15839) find
  63. Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. 2023a. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308.15126. — [arXiv:2308.15126](https://arxiv.org/abs/2308.15126) find
  64. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023b. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160.
  65. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the…
  66. Yujun Wang, Jinhe Bi, Yunpu Ma, and Soeren Pirk. 2025b. Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm. arXiv preprint arXiv:2506.14766. — [arXiv:2506.14766](https://arxiv.org/abs/2506.14766) find
  67. Zehao Wang, Xinpeng Liu, Xiaoqian Wu, Yudonglin Zhang, Zhou Fang, Yifan Fang, Junfu Pu, Cewu Lu, and Yong-Lu Li. 2024c. Verb mirage: Unveiling and assessing verb concept hallucinations in multimodal large language models. arXiv preprint arXiv:2412.04939. — [arXiv:2412.04939](https://arxiv.org/abs/2412.04939) find
  68. Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E Gonzalez, Trevor Darrell, and David M Chan. 2025b. Generate, but verify: Reducing hallucination in vision-language models with retrospective resampling. arXiv preprint arXiv:2504.13169. find
  69. Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Hao Jiang, Fei Wu, and Linchao Zhu. 2024. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. arXiv preprint arXiv:2404.14233. — [arXiv:2404.14233](https://arxiv.org/abs/2404.14233) find
  70. Chunzhao Xie, Tongxuan Liu, Lei Jiang, Yuting Zeng, Yunheng Shen, Weizhe Huang, Jing Li, Xiaohua Xu, et al. 2025. Tarac: Mitigating hallucination in lvlms via temporal attention real-time accumulative connection. arXiv preprint arXiv:2504.04099. — [arXiv:2504.04099](https://arxiv.org/abs/2504.04099) find
  71. Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024a. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926. — [arXiv:2410.15926](https://arxiv.org/abs/2410.15926) find
  72. Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024b. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926. — [arXiv:2410.15926](https://arxiv.org/abs/2410.15926) find
  73. Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. 2024. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. arXiv preprint arXiv:2412.13817. — [arXiv:2412.13817](https://arxiv.org/abs/2412.13817) find
  74. Hao Yin, Guangzong Si, and Zilei Wang. 2025. Clearsight: Visual signal enhancement for object hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2503.13107. — [arXiv:2503.13107](https://arxiv.org/abs/2503.13107) find
  75. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.
  76. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
  77. Jinfeng Wei and Xiaofeng Zhang. 2024. Dopra: Decoding over-accumulation penalization and re-allocation in specific weighting layer. Proceedings of the 32nd ACM International Conference on Multimedia. find
  78. Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406.15765. — [arXiv:2406.15765](https://arxiv.org/abs/2406.15765) find
  79. Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. 2024. Ritual: Random image transformations as a universal anti-hallucination lever in lvlms. arXiv preprint arXiv:2405.17821. — [arXiv:2405.17821](https://arxiv.org/abs/2405.17821) find
  80. Liu Yuan, Duan Haodong, Zhang Yuanhan, Li Bo, Zhang Songyang, and Zhao Wangbo. 2023. Mmbench:is your multi-modal model an all-around player. arXiv:2307.06281.
  81. Jiarui Wu, Zhuo Liu, and Hangfeng He. 2025a. Mitigating hallucinations in multimodal spatial relations through constraint-aware prompting. arXiv preprint arXiv:2502.08317. find
  82. Zihao Yue, Liang Zhang, and Qin Jin. 2024. Less is more: Mitigating multimodal hallucination from an eos decision perspective. The 62nd Annual Meeting of the Association for Computational Linguistics. find
  83. Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. 2023. Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption. arXiv preprint arXiv:2310.01779. — [arXiv:2310.01779](https://arxiv.org/abs/2310.01779) find
  84. Haozhe Zhao, Shuzheng Si, Liang Chen, Yichi Zhang, Maosong Sun, Mingjia Zhang, and Baobao Chang. 2024. Looking beyond text: Reducing language bias in large vision-language models via multimodal dual-attention and soft-image guidance. arXiv preprint arXiv:2411.14279. find
  85. Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia Sycara, and Yaqi Xie. 2025a. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2502.06130… — [arXiv:2502.06130](https://arxiv.org/abs/2502.06130) find
  86. Qiyan Zhao, Xiaofeng Zhang, Shuochen Chang, Qianyu Chen, Xiaosong Yuan, Xuhang Chen, Luoqi Liu, Jiajun Zhang, Xu-Yao Zhang, and Da-Han Wang. 2026. Context tokens are anchors: Understanding the repetition curse in dmllms from an information flow perspective. In The Fourteenth International Conference… find
  87. Gengyuan Zhang, Jinhe Bi, Jindong Gu, Yanyu Chen, and Volker Tresp. 2023. Spot! revisiting video-language models for event understanding. arXiv preprint arXiv:2311.12919. — [arXiv:2311.12919](https://arxiv.org/abs/2311.12919) find
  88. Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024a. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms. arXiv preprint arXiv:2411.09968. — [arXiv:2411.09968](https://arxiv.org/abs/2411.09968) find
  89. Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. 2024b. From redundancy to relevance: Enhancing explainability in multimodal large language models. arXiv preprint arXiv:2406.06579. — [arXiv:2406.06579](https://arxiv.org/abs/2406.06579) find
  90. Xiaofeng Zhang, Fanshuo Zeng, and Chaochen Gu. 2024c. Simignore: Exploring and enhancing multimodal large model complex reasoning via similarity computation. Neural Networks, page 107059. find
  91. Xiaofeng Zhang, Fanshuo Zeng, and Chaochen Gu. 2025b. Simignore: Exploring and enhancing multimodal large model complex reasoning via similarity computation. Neural Networks, 184:107059. find
  92. Qiyan Zhao, Xiaofeng Zhang, yiheng Li, Yun Xing, Yuan Xiaosong, Feilong Tang, Sinan Fan, Xuhang Chen, Xuyao Zhang, and Dahan Wang. 2025. Mca-llava: Manhattan causal attention for reducing hallucination in large vision-language models. The 33rd ACM International Conference on Multimedia. find
  93. Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, and Lichao Sun. 2024. Thinking before looking: Improving multimodal llm reasoning via mitigating visual hallucination. arXiv preprint arXiv:2411.12591. — [arXiv:2411.12591](https://arxiv.org/abs/2411.12591) find
  94. Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2024. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality. arXiv preprint arXiv:2410.04780. — [arXiv:2410.04780](https://arxiv.org/abs/2410.04780) find
  95. Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2025. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality. In The Thirteenth International Conference on Learning Representations. find
  96. Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. 2024d. Enhancing multimodal large language models complex reason via similarity computation. arXiv preprint arXiv:2412.09817. — [arXiv:2412.09817](https://arxiv.org/abs/2412.09817) find
  97. Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754.
  98. Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. 2025c. Enhancing multimodal large language models complex reason via similarity computation. Proceedings of the AAAI Conference on Artificial Intelligence. find
  99. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
  100. Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Jiawei Cao, Hao Cheng, and Kaijie Wu. 2026a. What drives attention sinks? a study of massive activations and rotational positional encoding in large vision–language models. Information Processing & Management, 63(2):104431. find
  101. Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Xiaosong Yuan, Qiyan Zhao, Jiawei Cao, Feilong Tang, Sinan Fan, Yaomin Shen, Chen Shen, et al. 2026b. Hallucination begins where saliency drops. In The Fourteenth International Conference on Learning Representations. find
  102. Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476. — [arXiv:2402.18476](https://arxiv.org/abs/2402.18476) find
  103. Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. 2024. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2410.03577. — [arXiv:2410.03577](https://arxiv.org/abs/2410.03577) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-tokenpenalty-alleviating-attention-sinks-positional/