Elamur Research⌕
☰
home / papers / 2023 / Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis

Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis

Summary

Emotion arcs trace how sentiment toward a topic, or within a story or population, changes over time. They are widely used, but their accuracy has rarely been checked because true arcs are hard to obtain. The authors address this by building gold arcs from datasets whose individual instances carry human valence labels. They sort the instances, slide a fixed-size bin across them one instance at a time, and average the human scores in each bin. Automatically produced arcs are z-score normalized and compared with these gold arcs using Spearman correlation and RMSE. The study pits a simple Lexicon-Only (LexO) approach against supervised Machine-Learning (ML) classifiers on 18 datasets in nine languages. LexO is much weaker at labeling single instances, yet once bins hold a few hundred instances its arcs nearly match the gold ones. The same holds when English lexicons are machine-translated into Arabic, Spanish, and six African languages. This means cheap, interpretable arc generation is feasible for under-resourced languages.

Key points

  • Data: 18 sentiment-labeled datasets covering two English movie-review sets, four SemEval 2014 English sets, six SemEval 2018 sets (English, Arabic, Spanish), and six AfriSenti (SemEval 2023) sets in Amharic, Hausa, Igbo, Kinyarwanda, Swahili, and Yoruba.
  • At bin size 100, both LexO and ML arcs correlate with gold valence arcs above 0.98. ML holds a clear advantage only for very small bins (fewer than 50 instances).
  • Arcs built with automatically translated English lexicons reach high correlations at bin sizes of a few hundred. Results tend to be better for languages more similar to English.
  • On more dynamic text streams with varied peak heights and widths, LexO remains robust. Small bins, however, can overshoot or miss peaks and troughs. In that setting, choices such as treating OOV words as neutral and using real-valued lexicons matter.
  • The work is limited to valence/sentiment arcs. Code and resources are available at github.com/dteodore/EmotionArcs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.271 · arXiv 2306.02213

@inproceedings{teodorescu2023evaluating,
  title = {{Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis}},
  author = {Teodorescu, Daniela and Mohammad, Saif},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.271},
  eprint = {2306.02213},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.271}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Our research interest is to study emotions at an aggregate/group level. This has applications in determining public policy (e.g., pandemic-response initiatives) and commerce (understanding attitudes towards products). However, emotions are complex, private, and central to an individual’s experience.… find
  2. Atsushi Ashida, Masataka Tokumaru, and Tomoko Ko-jiri. 2021. Characters’ emotion design support system for writing novels based on story arcs of target readers. Information and Technology in Education and Learning, 1:Reg–p005. find
  3. Sriharsh Bhyravajjula, Ujwal Narayan, and Manish Shrivastava. 2022. Marcus: An event-centric nlp pipeline that generates character arcs from narratives. In Text2Story@ECIR. find
  4. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. In Seventh international AAAI conference on weblogs and social media, pages 128–137. find
  5. Marco Del Vecchio, Alexander Kharlamov, Glenn Parry, and Ganna Pogrebna. 2018. The Data Science of Hollywood: Using Emotional Arcs of Movies to Drive Business Model Innovation in Entertainment Industries. arXiv e-prints, page arXiv:1807.02221. — [arXiv:1807.02221](https://arxiv.org/abs/1807.02221) find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Sharath Chandra Guntuku, Rachelle Schneider, Arthur Pelullo, Jami Young, Vivien Wong, Lyle Ungar, Daniel Polsky, Kevin G Volpp, and Raina Merchant. 2019. Studying expressions of loneliness in individuals using Twitter: an observational study. BMJ open, 9(11):e030355. find
  8. Will E. Hipson and Saif M. Mohammad. 2021. Emotion dynamics in movie dialogues. PLOS ONE, 16(9):1– 19. find
  9. Alexander Hogenboom, Bas Heerschop, Flavius Frasincar, Uzay Kaymak, and Franciska de Jong. 2014. Multi-lingual support for lexicon-based sentiment analysis guided by semantics. Decision Support Systems, 62:43–53. find
  10. Tao Hu, Siqin Wang, Wei Luo, Mengxi Zhang, Xiao Huang, Yingwei Yan, Regina Liu, Kelly Ly, Viraj Kacker, Bing She, et al. 2021. Revealing public opinion towards covid-19 vaccines with twitter data in the united states: spatiotemporal perspective. Journal of Medical Internet Research, 23(9):e30854. find
  11. Evgeny Kim, Sebastian Padó, and Roman Klinger. 2017. Investigating the relationship between literary genres and emotional plot development. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 17–26, Vancouv… find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. find
  13. Saif Mohammad. 2011. From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 105–114, Portland, OR, USA. Association for Computational Lin… find
  14. Saif Mohammad. 2023. Best practices in the creation and use of emotion lexicons. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1825–1836, Dubrovnik, Croatia. Association for Computational Linguistics. find
  15. Saif M. Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of The Annual Conference of the Association for Computational Linguistics (ACL), Melbourne, Australia. find our card
  16. Saif M. Mohammad. 2020. Practical and ethical considerations in the effective use of emotion and sentiment lexicons. find
  17. Saif M. Mohammad. 2021. Sentiment analysis: Automatically detecting valence, emotions, and other affectual states from text. In Herb Meiselman, editor, Emotion Measurement (Second Edition). Elsevier. find
  18. Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. To Appear in Computational Linguistics. find
  19. Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA. find our card
  20. Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. J. Artif. Int. Res., 55(1):95–130. find
  21. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  22. Shamsuddeen Muhammad, Seid Yimam, Idris Abdulmumin, Ibrahim Ahmad, David Adelani, Sebastian Ruder, Nedjma Ousidhoum, Saminu Aliyu, Abinew Ayele, Meriem Beloucif, Bello Bello, and Saif M. Mohammad. 2023. find
  23. SemEval 2023 task 12: Sentiment analysis for african languages (afrisenti-semeval). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Toronto, Canada. find
  24. Shamsuddeen Hassan Muhammad, David Adelani, Sebastian Ruder, Ibrahim Sa’id Ahmad, Idris Abdulmumin, Shehu Bello Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahuddeen Abdullahi, Anuoluwapo Aremu, Alipio Jeorge, and Pavel Brazdil. 2022. Naijasenti: A nigerian twitter sentiment corpus for… find our card
  25. Emily Öhman. 2021. The validity of lexicon-based sentiment analysis in interdisciplinary research. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pages 7–12, NIT Silchar, India. NLP Association of India (NLPAI). find
  26. Andrew J. Reagan, Lewis Mitchell, Dilan Kiley, Christopher M. Danforth, and Peter S. Dodds. 2016. The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science, 5(1):1–12. Copyright -EPJ Data Science is a copyright of Springer, 2016; Last updated - 2017-02-06. find
  27. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland. Association for Computational Linguistics. find
  28. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  29. Swapna Somasundaran, Xianyang Chen, and Michael Flor. 2020. Emotion arcs of student narratives. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 97–107, Online. Association for Computational Linguistics. find
  30. Charles Spearman. 1987. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441–471. find
  31. Daniela Teodorescu, Tiffany Cheng, Alona Fyshe, and Saif M. Mohammad. 2023a. Language and mental health: Measures of emotion dynamics from text as linguistic biosocial markers. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  32. Daniela Teodorescu, Alona Fyshe, and Saif Mohammad. 2023b. Utterance emotion dynamics in children’s poems: Emotional changes across age. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 401–415, Toronto, Canada. Association fo… find
  33. Krishnapriya Vishnubhotla and Saif M. Mohammad. 2022. Tweet emotion dynamics: Emotion word usage in tweets from us and canada. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. find
  34. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find our card
  35. Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020. Exploring find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/teodorescu-2023-evaluating-emotion-arcs-across-languages/