Elamur Research
☰
home / papers / 2018 / Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words

Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1017.pdf

Summary

Popular valence–arousal–dominance (VAD) lexicons, such as ANEW and the Warriner et al. (2013) norms, were built with rating scales. That method brings known problems: annotators disagree with each other and with themselves, they favour certain parts of the scale, and the granularity is fixed. This paper presents the NRC VAD Lexicon, which gives crowdsourced scores on all three dimensions for 20,007 common English terms. These include emotion-related words and tweet vocabulary such as emoticons.

Instead of rating single words, annotators used Best–Worst Scaling. They saw four words at a time and picked the highest and the lowest on one dimension. Each word's real-valued score from 0 to 1 comes from how often it was chosen best or worst.

The resulting scores show much higher split-half reliability than earlier lexicons. The lexicon also covers roughly 40% more words than the largest previous manual VAD resource. Optional demographic questionnaires showed significant differences in agreement linked to age, gender and personality. The lexicon is free for research and non-commercial use in sentiment and emotion analysis, evaluation of automatic methods, and psychology.

Key points

  • Term selection: the vocabulary is the union of several sources:
    • the NRC Emotion Lexicon
    • General Inquirer positive/negative lists
    • all 1,061 ANEW terms
    • all 13,915 Warriner et al. terms
    • Roget's Thesaurus emotion categories
    • about 1,000 frequent terms from the Hashtag Emotion Corpus (HEC)
  • Annotation design: for each dimension, 40,014 random 4-tuples (2N) were generated on CrowdFlower. Each word appeared in eight tuples, and each tuple received at least 6 annotations. In total there were 778,085 best–worst judgments from roughly 1,000 annotators per dimension.
  • Reliability: split-half reliability is r = 0.95 for valence, 0.90 for arousal and about 0.90–0.91 for dominance (the paper reports both values). These are clearly above the Warriner et al. lexicon.
  • Demographic findings:
    • Annotators over 35 were more consistent than younger ones.
    • Men agreed more with each other on valence and dominance, while women agreed more on arousal.
    • Extroverts showed higher in-group agreement than introverts.
    • A sanity check splitting annotators by odd versus even worker IDs showed no differences.
  • Availability and uses: the lexicon is distributed via saifmohammad.com. It is intended for use alongside the NRC Emotion Lexicon and the NRC Affect Intensity Lexicon, and as gold scores for evaluating automatic VAD prediction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

187 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1017

@inproceedings{mohammad2018obtaining,
  title = {{Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words}},
  author = {Mohammad, Saif},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1017},
  url = {https://doi.org/10.18653/v1/p18-1017}
}

References

40 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, California. find
  2. Iris Bakker, Theo van der Voordt, Peter Vink, and Jan de Boon. 2014. Pleasure, arousal, dominance: Mehrabian and russell revisited. Current Psychology, 33(3):405–421. find
  3. Ai-Min Bao and Dick F Swaab. 2011. Sexual differentiation of the human brain: relation to gender identity, sexual orientation and neuropsychiatric disorders. Frontiers in neuroendocrinology, 32(2):214– 226. find
  4. Svetlana Kiritchenko and Saif M. Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. find
  5. Murray R Barrick and Michael K Mount. 1991. The big five personality dimensions and job performance: a meta-analysis. Personnel psychology, 44(1):1–26. find
  6. Peer M Lewinsohn, Ian H Gotlib, Mark Lewinsohn, John R Seeley, and Nicholas B Allen. 1998. Gender differences in anxiety disorders and anxiety symptoms in adolescents. Journal of abnormal psychology, 107(1):109. find
  7. Hans Baumgartner and Jan-Benedict E.M. Steenkamp. 2001. Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2):143–156. find
  8. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. find
  9. Margaret M Bradley and Peter J Lang. 1999. Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, The Center for Research in Psychophysiology, University of Florida. find
  10. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  11. Carmen P McLean, Anu Asnaani, Brett T Litz, and Stefan G Hofmann. 2011. Gender differences in anxiety disorders: prevalence, course of illness, comorbidity and burden of illness. Journal of psychiatric research, 45(8):1027–1035. find
  12. Phillip M Chmielewski, Leyan OL Fernandes, Cindy M Yee, and Gregory A Miller. 1995. Ethnicity and gender in scales of psychosis proneness and mood disorders. Journal of Abnormal Psychology, 104(3):464. find
  13. Saif Mohammad. 2012. #Emotional Tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM), pages 246– 255, Montréal, Canada. find
  14. Steven H. Cohen. 2003. Maximum difference scaling: Improved measures of importance and preference for segmentation. Sawtooth Software, Inc. find
  15. Saif M. Mohammad. 2018. Word affect intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. find
  16. Herbert Aron David. 1963. The method of paired comparisons. Hafner Publishing Company, New York. find
  17. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, pages 178–201. Edward Elgar Publishing. find
  18. Saif M. Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Copenhagen, Denmark. find
  19. Eric O Johnson, Thomas Roth, Lonni Schultz, and Naomi Breslau. 2006. Epidemiology of dsm-iv insomnia in adolescence: lifetime prevalence, chronic-ity, and an emergent gender difference. Pediatrics, 117(2):e247–e256. find
  20. Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA. find our card
  21. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics, Atlanta, GA, USA. find
  22. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301–326. find
  23. Saif M. Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. find
  24. David Jurgens, Saif M. Mohammad, Peter Turney, and Keith Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation, pages 356–364, Montréal, Canada. find
  25. Changhua Yang, Kevin Hsin-Yih Lin, and Hsin-Hsi Chen. 2007. Building emotion lexicon from weblog corpora. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 133–136. find
  26. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  27. Agnes Moors, Jan De Houwer, Dirk Hermans, Sabine Wanmaker, Kevin Van Schie, Anne-Laura Van Harmelen, Maarten De Schryver, Jeffrey De Winne, and Marc Brysbaert. 2013. Norms of valence, arousal, dominance, and age of acquisition for 4,300 dutch words. Behavior research methods, 45(1):169–177. find
  28. Liang-Chih Yu, Jin Wang, K Robert Lai, and Xue-jie Zhang. 2015. Predicting valence-arousal ratings of words using a weighted graph method. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 788–793. find
  29. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. find
  30. C.E. Osgood, Suci G., and P. Tannenbaum. 1957. The measurement of meaning. University of Illinois Press. find
  31. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1(3):3–33. find
  32. Stanley Presser and Howard Schuman. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE Publications, Inc. find
  33. Jaime Redondo, Isabel Fraga, Isabel Padrón, and Montserrat Comesaña. 2007. The spanish adaptation of anew (affective norms for english words). Behavior research methods, 39(3):600–605. find
  34. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161. find
  35. James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review, 110(1):145. find
  36. Jacopo Staiano and Marco Guerini. 2014. Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605. ↗
  37. Philip Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. find
  38. Louis L. Thurstone. 1927. A law of comparative judgment. Psychological review, 34(4):273. find
  39. Melissa LH Võ, Markus Conrad, Lars Kuchinke, Karolina Urton, Markus J Hofmann, and Arthur M Jacobs. 2009. The berlin affective word list reloaded (bawl-r). Behavior research methods, 41(2):534– 538. find
  40. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4):1191–1207. find