Elamur Research⌕
☰
home / papers / 2016 / Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling

Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling

Paper · 2016 · Svetlana Kiritchenko and Saif M. Mohammad

Summary

Fine-grained, real-valued sentiment scores for words are more informative than simple positive/negative labels. However, asking annotators to give numeric ratings directly produces inconsistent results, and comparing every pair of terms grows quadratically with vocabulary size. Kiritchenko and Mohammad apply Best–Worst Scaling (BWS) through crowdsourcing on CrowdFlower. Annotators see four terms at a time and mark the most positive and the most negative one. A simple counting procedure turns these choices into scores between -1 and +1. With this method the authors built sentiment composition lexicons for three domains: English Twitter, Arabic Twitter, and general English with modifier phrases. All three lexicons were used in SemEval shared tasks. When the annotations are split into two independent groups, the resulting term rankings are almost identical. Two or three judgments per question are already enough for reliable scores. The authors also borrow the psychophysics notion of a just-noticeable difference and estimate the smallest sentiment difference that native speakers can perceive, which is about 0.08. This gives a principled threshold for studying how phrases compose sentiment.

Key points

  • Three lexicons were created:
    • SemEval-2015 English Twitter Lexicon: 1,515 terms, including misspellings, creative spellings, hashtags, emoticons and negated expressions.
    • SemEval-2016 Arabic Twitter Lexicon: 1,367 terms.
    • SemEval-2016 General English Sentiment Modifiers Lexicon (SCL-NMA): 1,621 Osgood words from General Inquirer, plus 1,586 phrases built with negators, modals and degree adverbs.
  • The setup used 2 × n four-term questions for n terms, each judged by ten respondents. A term's score is the share of times it was picked as most positive minus the share of times it was picked as most negative.
  • Agreement with the majority answer was 80–82%. Split-half reliability gave Spearman and Pearson correlations of at least 0.98 for every lexicon.
  • Scores from only 2 or 3 annotations per question correlated at 0.98 and 0.99 with the full ten-annotation scores. Conventional rating scales are reported to need 15–20 responses.
  • Pairwise agreement rises past 90% once two terms differ by about 0.4. The least perceptible difference was estimated at 0.069 (SCL-NMA), 0.080 (English Twitter) and 0.087 (Arabic Twitter).

Interesting ideas

  • I1. Comparisons instead of absolute ratings. Annotators pick extremes within small groups rather than assigning numbers. This avoids each person having their own internal scale, so it could suit any graded property, such as formality, offensiveness or regional usage, in Ukrainian datasets.
  • I2. Balanced tuple design. Each term and each pair of terms should appear in roughly equal numbers of four-term sets. Then about 1.5–2 × n questions give every term several independent assessments, so the annotation budget stays linear in the number of items.
  • I3. Split-half reproducibility as a quality metric. Randomly divide the responses, rebuild the ranking from each half, and correlate the two. This tests how stable the aggregated result is, which is more meaningful than raw per-item agreement.
  • I4. Learning the needed number of annotators from your own data. Subsample k responses per question and compare the resulting scores with the full set. This shows how many annotators are actually needed, which can save crowdsourcing costs on future projects.
  • I5. A "just-noticeable difference" threshold. Find the score gap at which pairwise human preference becomes reliably better than chance, using a confidence lower bound above 50%. Using it as a significance cutoff gives a principled way to decide whether a change, such as adding a modifier, really shifts a score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Academies of Sciences, Engineering, and MedicineWashington · United States11 works · 2,893 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 608 works

Cite

Cite as Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1095

@inproceedings{kiritchenko2016capturing,
  title = {{Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling}},
  author = {Svetlana Kiritchenko and Saif M. Mohammad},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1095},
  url = {https://doi.org/10.18653/v1/n16-1095}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Margaret M Bradley and Peter J Lang. 1999. Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, The Center for Research in Psychophysiology, University of Florida. find
  2. Steven H. Cohen. 2003. Maximum difference scaling: Improved measures of importance and preference for segmentation. find
  3. Sawtooth Software, Inc. Andrea Esuli and Fabrizio Sebastiani. 2006. SENTI-WORDNET: A publicly available lexical resource for opinion mining. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC), pages 417–422. find
  4. Gustav Fechner. 1966. Elements of psychophysics. Vol. find
  5. I. New York: Holt, Rinehart and Winston. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, pages 178–201. find
  6. Edward Elgar Publishing. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the Annual Conference of the North American Chapter of the ACL (NAACL), pages 1183–1191.
  7. Emily Grijalva, Daniel A. Newman, Louis Tay, M. Brent Donnellan, P.D. Harms, Richard W. Robins, and Taiyi Yan. 2015. Gender differences in narcissism: A meta-analytic review. Psychological bulletin, 141(2):261– 310. find
  8. Marcus Hartner. 2013. The lingering after-effects in the reader’s mind – an investigation into the affective dimension of literary reading. Journal of Literary Theory Online. find
  9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 168– 177, New York, NY, USA. find
  10. Svetlana Kiritchenko and Saif M. Mohammad. 2016. The effect of negators, modals, and degree adverbs on sentiment composition. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA). find
  11. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. find
  12. Svetlana Kiritchenko, Saif M. Mohammad, and Mohammad Salameh. 2016. SemEval-2016 Task 7: Determining sentiment intensity of English and Arabic phrases. In Proceedings of the International Workshop on Semantic Evaluation (SemEval), San Diego, California, June. find
  13. Jochen Kleres. 2011. Emotions and narrative analysis: A methodological approach. Journal for the Theory of Social Behaviour, 41(2):182–202. find
  14. Jordan J. Louviere and George G. Woodworth. 1990. Best-worst analysis. Working Paper. Department of Marketing and Economic Analysis, University of Alberta. find
  15. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. find
  16. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301–326. find
  17. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  18. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  19. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2016b. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, Submitted. find
  20. Saif M Mohammad. 2012. From once upon a time to happily ever after: Tracking emotions in mail and books. Decision Support Systems, 53(4):730–741. find
  21. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. find
  22. Sawtooth Software, Inc. Charles E Osgood, George J Suci, and Percy Tannenbaum. 1957. The measurement of meaning. University of Illinois Press. find
  23. Maria Pontiki, Harris Papageorgiou, Dimitrios Galanis, Ion Androutsopoulos, John Pavlopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval), Dublin, Ireland.
  24. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 Task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval), pages 73–80, Dublin, Ireland, August.
  25. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment analysis in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval), pages 450– 462, Denver, Colorado. Philip Stone, Dexter…
  26. The MIT Press. Duyu Tang, Furu Wei, Bing Qin, Ming Zhou, and Ting Liu. 2014. Building large-scale Twitter-specific sentiment lexicon: A representation learning approach. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 172–182. find
  27. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4):1191–1207. find
  28. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Joint Conference on HLT and EMNLP, pages 347–354, Stroudsburg, PA, USA.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kiritchenko-2016-capturing-reliable-fine-grained-sentiment/