Elamur Research⌕
☰
home / papers / 2025 / Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications

Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications

Paper · 2025 · Bushra Asseri, Estabraq Abdelaziz, Maha Al Mogren et al.

Summary

The paper asks how well general-purpose multimodal LLMs can read the emotions shown in illustrations from Arabic children's books. This matters because emotion-aware educational tools for Arabic-speaking children are scarce. The authors built a small benchmark of 75 illustrations from seven Arabic storybooks. Four native Arabic-speaking annotators labelled each image through a consensus procedure, using Plutchik's eight basic emotions plus a neutral class. GPT-4o and Gemini 1.5 Pro were each tested with zero-shot, few-shot and chain-of-thought prompts. Scoring used macro F1 and Cohen's kappa against the human labels. GPT-4o was ahead in every condition, peaking with chain-of-thought, while few-shot prompting tended to hurt. Error analysis found that mistakes were systematic rather than random: most were swaps between positive and negative feelings, and neutral or culturally loaded scenes caused trouble. Exploratory runs widened the label set and cropped single characters out of their scenes. The authors argue that emotions in children's stories serve a teaching role within the plot, so recognition needs to keep the cultural and narrative context.

Key points

  • Dataset: 75 images from seven Arabic storybooks (at least ten per book), labelled with Plutchik's taxonomy plus neutral by four native-speaker annotators.
  • GPT-4o reached macro F1 of 57–59% (best 59% with chain-of-thought); Gemini 1.5 Pro reached 32–43%.
  • Agreement with humans was moderate for GPT-4o (κ = 0.46–0.56) and fair for Gemini (κ = 0.31–0.37).
  • Valence inversions, where a positive emotion is read as negative or the reverse, made up 60.7% of misclassifications.
  • Labels were strongly skewed: happiness covered 30 of 75 images (40%), anger had 2 images and disgust only 1. The authors recommend zero-shot or chain-of-thought prompting with GPT-4o and advise against few-shot.

Interesting ideas

  • I1. Grouping errors by type, especially polarity flips, shows a structured weakness that a single F1 number would hide.
  • I2. Cutting a character out of its scene is a cheap ablation that shows how much a judgment depends on the surrounding story.
  • I3. Emotions in children's stories work as plot devices rather than standalone psychological labels, so they should be read from what happens in the story.
  • I4. No prompting strategy won for every model: one model gained from full context, the other from narrower focus or an explicit label scheme, and few-shot often hurt.
  • I5. Pairing macro F1 with kappa against human consensus gives a fairer picture when labels are very imbalanced.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When readers answer how a character in a fable feels, log valence-flip errors as their own category so that polarity bugs in the state logic stand out.
  • I2 (world): Run state inference on a single sentence and again with the full text world, as a negative control that shows whether event order and context really drive the inferred states.
  • I3 (frames): Attach emotion states to the frames that cause them (who did what to whom) instead of tagging them lexically, keeping each attribution explainable.
  • I4 (learn): Calibrate prompt style separately for each LLM teacher tier before trusting its silver labels, rather than assuming one template fits all.
  • I5 (learn): Report both metrics when evaluating rare frame or state classes so that a frequent majority class cannot inflate the scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alfaisal UniversityRiyadh · Saudi Arabia5 works · 23 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksComputer Vision and Pattern Recognition · 1,282 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 1,343 worksSentiment & emotion · 608 worksVision & language · 269 worksSemitic languages · 1,484 works

Cite

Cite as Bushra Asseri, Estabraq Abdelaziz, Maha Al Mogren, Tayef Alhefdhi and Areej Al-Wabil. 2025. Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications. arXiv:2506.18201.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3390/ai6090211 · arXiv 2506.18201

@misc{asseri2025deciphering,
  title = {{Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications}},
  author = {Bushra Asseri, Estabraq Abdelaziz, Maha {Al Mogren}, Tayef Alhefdhi and Areej Al-Wabil},
  year = {2025},
  doi = {10.3390/ai6090211},
  eprint = {2506.18201},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3390/ai6090211}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Susanne A. Denham, Hideko H. Bassett, and Katherine Zinsser. Early Childhood Teachers as Socializers of Young Children’s Emotional Competence. Early Childhood Education Journal, 40(3):137–143, June 2012. ↗
  2. Francisco Pons, Paul L Harris, and Marc de Rosnay. Emotion comprehension between 3 and 11 years: Developmental periods and hierarchical organization. European Journal of Developmental Psychology, 1(2):127–152, June 2004. Publisher: Routledge eprint: https://doi.org/10.1080/17405620344000022. ↗
  3. Xieling Chen, Haoran Xie, Di Zou, and Gwo-Jen Hwang. Application and theory gaps during the rise of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence, 1:100002, January 2020. ↗
  4. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 23716–23736, Red Hook, NY, USA, November 2022. Curran Associates Inc. ↗
  5. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July 2020. Association for Computational Linguistics.
  6. Nizar Y. Habash. Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies. Springer International Publishing, Cham, 2010. ↗
  7. Susanne A. Denham and Chavaughn Brown. Plays Nice With Others: Social–Emotional Learning and Academic Success. Early Education and Development, 21(5):652–680, October 2010. ↗
  8. Robert Plutchik. The Nature of Emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 89(4):344–350, 2001. Publisher: Sigma Xi, The Scientific Research Society. find
  9. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 24824–24837, Red Hook, NY, USA, November 2022. Curran Associates Inc.
  10. Emily M. Bender and Batya Friedman. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018. Place: Cambridge, MA Publisher: MIT Press. ↗
  11. Ali Al-Laith and Mamdouh Alenezi. Monitoring People’s Emotions and Symptoms from Arabic Tweets during the COVID-19 Pandemic. Information, 12(2):86, February 2021. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute. ↗
  12. L. S. Vygotsky. Mind in Society: Development of Higher Psychological Processes. Harvard University Press, 1978. find
  13. Jean Piaget. Intelligence and affectivity : their relationship during child development. Palo Alto, Calif. : Annual Reviews Inc., 1981. find
  14. David Matsumoto and Hyi Sung Hwang. Evidence for training the ability to read microexpressions of emotion. Motivation and Emotion, 35(2):181–191, June 2011. ↗
  15. Hassan Alhuzali and Sophia Ananiadou. SpanEmo: Casting Multi-label Emotion Classification as Span-prediction. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1573–1584, Online, April 2021. Association for Computational Linguistics. find
  16. Muhammad Abdul-Mageed and Lyle Ungar. EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 718–728, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  17. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August 2021. Association for Computational Linguistics. ↗
  18. Saif M. Mohammad and Peter D. Turney. Crowdsourcing a Word–Emotion Association Lexicon. Computational Intelligence, 29(3):436–465, 2013. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8640.2012.00460.x. ↗
  19. El Habib Nfaoui and Hanane Elfaik. Evaluating Arabic Emotion Recognition Task Using ChatGPT Models: A Comparative Analysis between Emotional Stimuli Prompt, Fine-Tuning, and In-Context Learning. Journal of Theoretical and Applied Electronic Commerce Research, 19(2):1118–1141, June 2024. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute. ↗
  20. Georgia Chronaki, Michael Wigelsworth, Marc D. Pell, and Sonja A. Kotz. The development of cross-cultural recognition of vocal emotion during childhood and adolescence. Scientific Reports, 8(1):8659, June 2018. Publisher: Nature Publishing Group. ↗
  21. Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, April 2023.
  22. Gemini Team. Gemini: A Family of Highly Capable Multimodal Models, December 2023. ↗
  23. Jezia Zakraoui, Moutaz Saleh, Somaya Al-Maadeed, and Jihad Mohamad AlJa’am. A study of children emotion and their performance while handwriting Arabic characters using a haptic device. Education and Information Technologies, 28(2):1783–1808, February 2023. ↗
  24. Joseph A. Durlak, Roger P. Weissberg, Allison B. Dymnicki, Rebecca D. Taylor, and Kriston B. Schellinger. The Impact of Enhancing Students’ Social and Emotional Learning: A Meta-Analysis of School-Based Universal Interventions. Child Development, 82(1):405–432, 2011. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8624.2010.01564.x. ↗
  25. Maha Jarallah Althobaiti. An open-source dataset for arabic fine-grained emotion recognition of online content amid COVID-19 pandemic. Data in Brief, 51:109745, December 2023. find
  26. Taghyeer Association. We Love Reading, Arabic Children’s Literature Collection, 2023. find
  27. James Russell. A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39:1161–1178, December 1980. ↗
  28. Dirk Hovy and Shannon L. Spruit. The Social Impact of Natural Language Processing. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany, August 2016. Association for Computational Linguistics. ↗
  29. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. January 2022. ↗
  30. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Ni Chhaya, and Alexander Gelbukh. DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation. January 2019. Pages: 164.
  31. Saif Mohammad and Svetlana Kiritchenko. Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). find
  32. Jonathan Posner, James A. Russell, and Bradley S. Peterson. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(3):715–734, 2005. ↗
  33. J. A. Russell and L. F. Barrett. Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. Journal of Personality and Social Psychology, 76(5):805–819, May 1999. ↗
  34. Jonathan Posner, James A. Russell, Andrew Gerber, Daniel Gorman, Tiziano Colibazzi, Shan Yu, Zhishun Wang, Alayar Kangarlu, Hongtu Zhu, and Bradley S. Peterson. The neurophysiological bases of emotion: An fMRI study of the affective circumplex using emotion-denoting words. Human Brain Mapping, 30(3):883–895, March 2008. ↗
  35. Rachael E. Jack, Oliver G. B. Garrod, Hui Yu, Roberto Caldara, and Philippe G. Schyns. Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19):7241–7244, May 2012. Publisher: Proceedings of the National Academy of Sciences. ↗
  36. Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychological Science in the Public Interest, 20(1):1–68, July 2019. Publisher: SAGE Publications Inc. ↗
  37. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A Survey on In-context Learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
  38. Lisa Feldman Barrett. The Conceptual Act Theory: A Précis. Emotion Review, 6(4):292–297, October 2014. Publisher: SAGE Publications. ↗
  39. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, New York, NY, USA, March 2021. Association for Computing Machinery. find
  40. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online, July 2020. Association for Computational Linguistics.
  41. Hillary Anger Elfenbein and Nalini Ambady. Cultural Similarity’s Consequences: A Distance Perspective on Cross-Cultural Differences in Emotion Recognition. Journal of Cross-Cultural Psychology, 34(1):92–110, January 2003. Publisher: SAGE Publications Inc. find
  42. Rosalind W. Picard. Affective computing: (526112012-054). 1997. Institution: American Psychological Association. find
  43. Marco Ribeiro, Sameer Singh, and Carlos Guestrin. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In John DeNero, Mark Finlayson, and Sravana Reddy, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 97–101, San Diego, California, June 2016. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asseri-2025-deciphering-emotions-children-storybooks-comparative/