Elamur Research⌕
☰
home / papers / 2026 / AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis

AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis

Paper · 2026 · Dong She, Xianrong Yao, Liqun Chen et al.

Summary

Existing emotion benchmarks for vision-language models mostly check whether a model can choose the right emotion label for an image. AICA-Bench tests a broader skill set across three tasks:

  • Emotion Understanding (EU): recognizing the emotion people in an image express and predicting the emotion a viewer would feel.
  • Emotion Reasoning (ER): explaining why an image evokes a given emotion.
  • Emotion-guided Content Generation (EGCG): writing a description that fits a target emotion.

The authors built the benchmark from nine public affective image datasets. Images were filtered automatically and then checked by people, and GPT-4o generated the task instructions. EU is scored with weighted F1. Open-ended answers are rated by an automatic assessor on emotion alignment, descriptiveness and, for ER, causal soundness.

Testing 23 open and closed VLMs zero-shot exposed two recurring failures. Models confuse high- and low-arousal emotions (intensity hallucination), and their free-text answers tend to be generic. To address these, the authors propose Grounded Affective Tree (GAT) Prompting, which needs no training. It combines visual scaffolding with tree-structured affective reasoning and grounded clue construction. GAT improved all three tasks for every tested model, giving a reusable baseline for affective multimodal research.

Key points

  • Scale: 8,086 images from EmoSet, Emotion6, FI, FlickrLDL, TwitterLDL, FindingEmo, EMOTIC (1,000 each), Artphoto (806) and Abstract (280), yielding 18,124 instructions. Twenty trained annotators removed NSFW and ambiguous images.
  • Assessor quality: the automatic assessor tracks human ratings closely, with Pearson 0.880 on ER and 0.900 on EGCG. GPT-4o, Gemini2.5-Pro and Qwen2.5VL-7B used as judges stay at or below 0.775.
  • GAT gains: average improvements are +6.15 pp on EU, +3.54 pp on ER and +3.96 pp on EGCG. For example, Gemini-2.5-Pro's EU score rises from 66.97 to 71.15.
  • Weakest EU baselines: LLaVA-1.6 models scored about 36–37 on EU, and GAT lifted LLaVA-1.6-7B to 51.05.
  • Ablation: for EU, the AffectToT reasoning module contributes more than visual scaffolding. For ER and EGCG, visual scaffolding and clue construction add up to the best results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 972 works

Cite

Cite as Dong She, Xianrong Yao, Liqun Chen, Jinghe Yu, Yang Gao and Zhanpeng Jin. 2026. AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.661

@inproceedings{she2026aica,
  title = {{AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis}},
  author = {Dong She, Xianrong Yao, Liqun Chen, Jinghe Yu, Yang Gao and Zhanpeng Jin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.661},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.661}
}

References

44 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  3. Sree Bhattacharyya and James Z. Wang. 2025. Evaluating vision-language models for emotion recognition. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1798–1820, Albuquerque, New Mexico. Association for Computational Linguistics. find
  4. Erik Cambria. 2016. Affective computing and sentiment analysis. IEEE Intelligent Systems, 31(2):102–107. promote vision-language coordination in large multimodal models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2886–2903, Abu Dhabi, UAE. Association for… find
  5. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. In NeurIPS. find
  6. Soumik Chakrabarti. 2020. Felzenszwalb segmentation: Python implementation of efficient graph-based image segmentation. https://github.com/ soumik12345/felzenszwalb_segmentation. Accessed: 2025-12-10. ↗
  7. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others. 2025. Expanding performance boundaries of open-source multi…
  8. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. Preprint, arXiv:2312.14238.
  9. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394.
  10. Lancheng Gao, Ziheng Jia, Yunhao Zeng, Wei Sun, Yiming Zhang, Wei Zhou, Guangtao Zhai, and Xiongkuo Min. 2025. Eemo-bench: A benchmark for multimodal large language models on image evoked emotion assessment. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25, page 7064–707… find
  11. He Hu, Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, and Laizhong Cui. 2025. Emobench-m: Benchmarking emotional intelligence for multimodal large language models. Preprint, arXiv:2502.04424. — [arXiv:2502.04424](https://arxiv.org/abs/2502.04424) find
  12. Ronak Kosti, Jose M. Alvarez, Adria Recasens, and Agata Lapedriza. 2017. Emotic: Emotions in context dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2309–2317. find
  13. Ronak Kosti, Jose M. Alvarez, Adria Recasens, and Agata Lapedriza. 2020. Context based emotion recognition using emotic dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(11):2755–2766. find
  14. Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. 2025. Scaffolding coordinates to Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408…
  15. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C.H. Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086.
  16. Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, and Jianhua Tao. 2025. AffectGPT: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. In Forty-second International Confer… find
  17. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
  19. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286– 26296.
  20. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge. find
  21. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
  22. Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2023b. Query-relevant images jailbreak large multi-modal models. ArXiv, abs/2311.17600. find
  23. Yexiang Liu, Zekun Li, Zhi Fang, Nan Xu, Ran He, and Tieniu Tan. 2025. Rethinking the role of prompting strategies in LLM test-time scaling: A perspective of probability theory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  24. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024c. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–Octob…
  25. Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. Preprint, arXiv:2405.20797. — [arXiv:2405.20797](https://arxiv.org/abs/2405.20797) find
  26. Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the 18th ACM International Conference on Multimedia, MM ’10, page 83–92, New York, NY, USA. Association for Computing Machinery. find
  27. Laurent Mertens, Elahe Yargholi, Hans Op de Beeck, Jan Van den Stock, and Joost Vennekens. 2024. Findingemo: An image dataset for emotion recognition in the wild. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  29. Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew Gallagher. 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 860–868. find
  30. Rosalind W. Picard. 1997. Affective Computing. The MIT Press. find
  31. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII, page 14…
  32. João Sedoc, Daniel Preoţiuc-Pietro, and Lyle Ungar. 2017. Predicting emotional word ratings using distributional representations and signed clustering. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 564–57… find
  33. Shezheng Song, Chengxiang He, Shasha Li, Shan Zhao, Chengyu Wang, Tianwei Yan, Xiaopeng Li, Qian Wan, Jun Ma, Jie Yu, and Xiaoguang Mao. 2024. Mosabench: Multi-object sentiment analysis benchmark for evaluating multimodal large language models understanding of complex image. Preprint, arXiv:2412.000… — [arXiv:2412.00060](https://arxiv.org/abs/2412.00060) find
  34. Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emirates. Association for…
  35. Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. 2024. How many are in this image a safety evaluation benchmark for vision llms. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2… find
  36. Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. 2023. What makes for good visual tokenizers for large language models? Preprint, arXiv:2305.12223. — [arXiv:2305.12223](https://arxiv.org/abs/2305.12223) find
  37. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  38. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2025. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders. In Forty-second International Conference on Machine Learning. find
  39. Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2023. Emoset: A large-scale visual emotion dataset with rich attributes. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20326– 20337. find
  40. Jufeng Yang, Dongyu She, and Ming Sun. 2017. Joint image emotion classification and distribution learning via deep convolutional neural network. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3266–3272. find
  41. Qu Yang, Mang Ye, and Bo Du. 2024. Emollm: Multimodal emotional understanding meets large language models. Preprint, arXiv:2406.16442. — [arXiv:2406.16442](https://arxiv.org/abs/2406.16442) find
  42. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems. find
  43. Y. Yao, T. Yu, A. Zhang, J. Liu, H. Wang, and X. Chen. 2025. Efficient gpt-4v level multimodal large language model for deployment on edge devices. Nature Communications, 16:5509. find
  44. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, and 4 others. 2024. Minicpmv: A gpt-4v level mllm on your phone. Preprint, arX…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/she-2026-aica-bench-holistically-examining-capabilities/