Elamur Research⌕
☰
home / papers / 2026 / AGIC: Attention-Guided Image Captioning to Improve Caption Relevance

AGIC: Attention-Guided Image Captioning to Improve Caption Relevance

Paper · 2026 · L. D. M. S. Sai Teja, Ashok Urlana and Pruthwik Mishra

Summary

Image captioning systems often produce vague descriptions that miss secondary objects or details. AGIC is a training-free remedy. A pretrained vision transformer first produces self-attention maps, which are averaged over heads and layers. These maps rescale the input image so that strongly attended regions are boosted by a factor (1 + k·attention). The modified image then goes to a captioning model, which decodes with a hybrid scheme: beam search combined with Top-k, Top-p and temperature-scaled sampling inside each beam. The authors evaluate on Flickr8k, Flickr30k and MSCOCO against zero-shot VLMs, unsupervised and supervised captioners. The strongest variant uses CLIP attention with BLIP2 generation. It matches or beats the baselines on most metrics, including the highest CIDEr and CLIPScore on both reported datasets, and it runs inference faster. The approach also helps visual question answering. It matters because it improves caption detail and relevance without paired training data or fine-tuning. It only needs access to model attention, so it works with open-source models.

Key points

  • The method needs no training. Amplification follows I_a = I_o·(1 + k·a), and k = 1 works best. Negative or larger k values lower scores, and the method is sensitive to this setting. Averaging attention over all layers beats using the first, middle, last or max layer.
  • On MSCOCO, AGIC + CLIP + BLIP2 reaches CIDEr 1.092 and CLIPScore 0.853. The strongest baseline, the supervised LSTNet, scores 0.991 and 0.834.
  • On Flickr8k, the same variant reaches CIDEr 0.786 and CLIPScore 0.851. LSTNet scores 0.623 and 0.781.
  • Baselines include BLIP2-opt-2.7b, LLaVA-1.5-7B, Qwen2.5-VL-7B-Instruct, Fuyu-8B, R²M, BRNN, LSTNet, AoANet, CapMAS and SPARC.
  • Human ratings on 500 samples (1–5 scale) average 4.04 for correctness, 3.64 for completeness and 4.26 for relevance, with ICC between 0.82 and 0.90. Omissions are the most common error type.
  • On a 500-image subset of VQA v2, overall accuracy rises from 0.532 (BLIP-2 alone) to 0.596. The method cannot be applied to closed models, because it depends on attention weights.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Institute Of Technology SilcharSilchar · India6 works · 72 citations🎓Sardar Vallabhbhai National Institute of Technology SuratSurat · India5 works · 4 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as L. D. M. S. Sai Teja, Ashok Urlana and Pruthwik Mishra. 2026. AGIC: Attention-Guided Image Captioning to Improve Caption Relevance. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.342

@inproceedings{teja2026agic,
  title = {{AGIC: Attention-Guided Image Captioning to Improve Caption Relevance}},
  author = {L. D. M. S. Sai Teja, Ashok Urlana and Pruthwik Mishra},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.342},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.342}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Israa Al Badarneh, Bassam H Hammo, and Omar Al-Kadi. 2025. An ensemble model with attention based mechanism for image captioning. Computers and Electrical Engineering, 123:110077. find
  2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer.
  3. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086. find
  4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. find
  5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  6. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
  7. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. 2023. Introducing our multimodal models. find
  8. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
  9. Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2019. Show, control and tell: A framework for generating controllable and grounded captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8307–8316. find
  10. Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578–10587. find
  11. Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2019. Unsupervised image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4125–4134. find
  12. Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. 2019. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10323–10332. find
  13. Dan Guo, Yang Wang, Peipei Song, and Meng Wang. 2020. Recurrent relational memory network for unsupervised image captioning. arXiv preprint arXiv:2006.13611. — [arXiv:2006.13611](https://arxiv.org/abs/2006.13611) find
  14. Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019. Image captioning: Transforming objects into words. Advances in neural information processing systems, 32. find
  15. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718.
  16. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853–899. find
  17. Ukyo Honda, Yoshitaka Ushiku, Atsushi Hashimoto, Taro Watanabe, and Yuji Matsumoto. 2021. Removing word-level spurious alignment between images and pseudo-captions in unsupervised image captioning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin… find
  18. Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019. Attention on attention for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4634–4643. find
  19. Mingi Jung, Saehyung Lee, Eunji Kim, and Sungroh Yoon. 2025. Visual attention never fades: Selective progressive attention recalibration for detailed image captioning in multimodal large language models. arXiv preprint arXiv:2502.01419. — [arXiv:2502.01419](https://arxiv.org/abs/2502.01419) find
  20. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128– 3137.
  21. Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155–163. find
  22. Iro Laina, Christian Rupprecht, and Nassir Navab. 2019. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7414–7424. find
  23. Saehyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, and Sungroh Yoon. 2024. Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint arXiv:2412.15484. — [arXiv:2412.15484](https://arxiv.org/abs/2412.15484) find
  24. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  25. Xu Liang, Chen Li, and Lihua Tian. 2024. Generative adversarial network for semi-supervised image captioning. Computer Vision and Image Understanding, 249:104199. find
  26. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  27. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
  28. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  29. Qiang Liu, Xinlong Chen, Yue Ding, Shizhen Xu, Shu Wu, and Liang Wang. 2025. Attention-guided self-reflection for zero-shot hallucination detection in large language models. arXiv preprint arXiv:2501.09997. find
  30. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7219–7228. find
  31. Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Yiyi Zhou, and Rongrong Ji. 2023. Towards local visual modeling for image captioning. Pattern Recognition, 138:109420. find
  32. Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10971– 10980. from natural language supervision. In International conference on machine learning, pages 87… find
  33. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.
  34. Zhan Shi, Hui Liu, and Xiaodan Zhu. 2021. Enhancing descriptive image captioning with natural language inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pap… find
  35. Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. 2020. Improving image captioning with better use of caption. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7454–7464, Online. Association for Computational Linguistics. find
  36. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  37. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048–2057. PMLR.
  38. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/teja-2026-agic-attention-guided-image-captioning/