Elamur Research⌕
☰
home / papers / 2026 / Evaluating Multimodal Large Language Model Narrative Interpretation through the Lens of Appraisal Theory

Evaluating Multimodal Large Language Model Narrative Interpretation through the Lens of Appraisal Theory

Summary

The paper asks whether multimodal large language models (MLLMs) read emotions in images the way people do under Appraisal Theory. In that theory, feelings come from judging a situation and then revising the judgment as more context arrives. The authors ran five MLLMs under three controlled conditions. In E1 the model sees only the cropped person. In E2 it sees the full scene around that person. In E3 it sees the full scene plus a narrative description that the same model wrote beforehand. Experiments use two public context-aware emotion datasets, Emotic and HECO, scored with Macro F1.

The results contradict the cognitive expectation. Showing the scene often lowers accuracy compared with the isolated person. The self-written narrative either hurts performance (HECO) or leaves it roughly unchanged (Emotic). The authors conclude that current MLLMs neither extract useful stories from visual surroundings nor exploit textual narratives when judging emotions. This exposes a gap between benchmark success and human-like contextual reasoning, and the setup offers a cognition-grounded way to evaluate affective AI.

Key points

  • Research questions: R1 asks whether visual context helps MLLM emotion recognition. R2 asks how well MLLMs interpret the narratives contained in images.
  • Datasets: Emotic has 7,279 samples, 26 classes and multi-label prediction. HECO has 4,000 samples, 6 classes and single-label prediction. Both are evaluated with Macro F1.
  • Finding F1: Full-image input (E2) frequently scores below person-only input (E1). Background complexity seems to distract the models rather than help them.
  • Finding F2: Adding a self-generated narrative (E3) is often worse than E1 on HECO and comparable on Emotic. Models appear unable to produce or use helpful narratives.
  • Positioning: The work is grounded in appraisal frameworks by Arnold, Lazarus (CMRT), Scherer (Component Process Model) and Ellsworth & Smith. It contrasts with benchmarks such as EIBench and DEEMO, and claims to be the first probe of MLLM narrative comprehension in visual tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3kybwvtko6uq

@inproceedings{teotia2026evaluating,
  title = {{Evaluating Multimodal Large Language Model Narrative Interpretation through the Lens of Appraisal Theory}},
  author = {Teotia, Jayant and Wang, Xiaowei and Zhang, Xulang and Mao, Rui and Cambria, Erik},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3kybwvtko6uq},
  url = {https://doi.org/10.63317/3kybwvtko6uq}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Magda Blondiau Arnold. 1960. Emotion and personality. Psychological Aspects, 1. find
  3. Erik Cambria, Rui Mao, Amir Hussain, Keith Oatley, and Geoffrey Hinton. 2026. Artificial intelligence as the fourth decentering revolution: From cosmic, biological, and psychological displacement to cognitive decentering. Cognitive Computation, 18(20):1–13. find
  4. Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. 2024. What is the visual cognition gap between humans and multimodal LLMs? arXiv preprint arXiv:2406.10424. — [arXiv:2406.10424](https://arxiv.org/abs/2406.10424) find
  5. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811. find our card
  6. Phoebe C Ellsworth and Craig A Smith. 1988. From appraisal to emotion: Differences among unpleasant feelings. Motivation and emotion, 12(3):271–302. find
  7. Chunxiao Fan, Jie Lin, Rui Mao, and Erik Cambria. 2024. Fusing pairwise modalities for emotion recognition in conversations. Information Fusion, 106:102306. find
  8. Xin Hong, Yuan Gong, Vidhyasaharan Sethu, and Ting Dang. 2024. AER-LLM: Ambiguity-aware emotion recognition leveraging large language models. arXiv preprint arXiv:2409.18339. — [arXiv:2409.18339](https://arxiv.org/abs/2409.18339) find
  9. Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. 2019. Context based emotion recognition using emotic dataset. IEEE transactions on pattern analysis and machine intelligence, 42(11):2755–2766. find
  10. Richard S Lazarus. 1991. Progress on a cognitive-motivational-relational theory of emotion. American psychologist, 46(8):819. find
  11. Deng Li, Bohao Xing, Xin Liu, Baiqiang Xia, Bihan Wen, and Heikki Kälviäinen. 2025. Deemo: De-identity multimodal emotion recognition and reasoning. arXiv preprint arXiv:2504.19549. — [arXiv:2504.19549](https://arxiv.org/abs/2504.19549) find
  12. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. find our card
  13. Yijiang Li, Qingying Gao, Haoran Sun, Haiyun Lyu, Dezhi Luo, and Hokin Deng. 2024. CogDe-velop2K: Reversed cognitive development in multimodal large language models. arXiv preprint arXiv:2410.10855. — [arXiv:2410.10855](https://arxiv.org/abs/2410.10855) find
  14. Yuxiang Lin, Jingdong Sun, Zhi-Qi Cheng, Jue Wang, Haomin Liang, Zebang Cheng, Yifei Dong, Jun-Yan He, Xiaojiang Peng, and Xian-Sheng Hua. 2025. Why we feel: Breaking boundaries in emotional reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognitio… find
  15. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916. find our card
  16. Hao Lu, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang, Kaishen Yuan, Bin Huang, Zitong Yu, et al. 2024. GPT as psychologist? preliminary evaluations for GPT-4v on visual affective computing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti… find
  17. Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. 2024. GPTEval: A survey on assessments of ChatGPT and GPT-4. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7844–7866, Torino, Ital… find
  18. Rui Mao, Mengshi Ge, Sooji Han, Wei Li, Kai He, Luyao Zhu, and Erik Cambria. 2025a. A survey on pragmatic processing techniques. Information Fusion, 114:102712. find
  19. Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2023. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE Transactions on Affective Computing, 14(3):1743–1753. find
  20. Rui Mao, Qian Liu, Xiao Li, Erik Cambria, and Amir Hussain. 2025b. Bridging minds and machines: Toward an integration of ai and cognitive science. arXiv preprint arXiv:2508.20674. — [arXiv:2508.20674](https://arxiv.org/abs/2508.20674) find
  21. Keane Ong, Wei Dai, Carol Li, Dewei Feng, Hengzhi Li, Jingyao Wu, Jiaee Cheong, Rui Mao, Gianmarco Mengaldo, Erik Cambria, and Paul Pu Liang. 2025. Human behavior atlas: Benchmarking unified psychological and social behavior understanding. find
  22. Klaus R Scherer. 1987. Toward a dynamic theory of emotion: The component process model of affective states. Geneva studies in Emotion and Communication, 1:1–98. find
  23. Klaus R Scherer, Angela Schorr, and Tom Johnstone. 2001. Appraisal Processes in Emotion: Theory, Methods, Research. Oxford University Press. find
  24. Norbert Schwarz. 1996. Feelings and phenomenal experiences. Social psychology: Handbook of basic principles/Guilford. find
  25. Jayant Teotia, Xulang Zhang, Rui Mao, and Erik Cambria. 2024. Evaluating vision language models in detecting learning engagement. In 2024 IEEE International Conference on Data Mining Workshops (ICDMW), pages 496–502. IEEE. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998– 6008. find
  27. Luwei Xiao, Rui Mao, Shuai Zhao, Qika Lin, Yanhao Jia, Liang He, and Erik Cambria. 2025. Exploring cognitive and aesthetic causality for multimodal aspect-based sentiment analysis. IEEE Transactions on Affective Computing. find
  28. Yunhe Xie and Rui Mao. 2025. PGIF: A personality-guided iterative feedback graph network for multimodal conversational emotion recognition. IEEE Transactions on Computational Social Systems, 12(5):3583–3595. find
  29. Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, and Börje F Karlsson. 2024. A survey on game playing agents and large models: Methods, applications, and challenges. arXiv preprint arXiv:2403.10249. — [arXiv:2403.10249](https://arxiv.org/abs/2403.10249) find
  30. Dingkang Yang, Shuai Huang, Shunli Wang, Yang Liu, Peng Zhai, Liuzhen Su, Mingcheng Li, and Lihua Zhang. 2022. Emotion recognition for multiple context awareness. In European Conference on Computer Vision, pages 144–162. Springer. find
  31. Dong Zhang, Yuansheng Ma, Linqin Li, Shoushan Li, Erik Cambria, and Guodong Zhou. 2026. Training-free and zero-shot regeneration for hallucination mitigation in MLLMs: Representation understanding perspective. Expert Systems with Applications, 309:131102. find
  32. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/teotia-2026-evaluating-multimodal-large-language-model/