Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases

SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases

Paper · 2016 · Svetlana Kiritchenko, Saif Mohammad and Mohammad Salameh

Summary

The paper describes a SemEval-2016 shared task on predicting a real-valued prior sentiment score between 0 and 1 for words and multi-word phrases. Systems only had to order terms correctly, so a more positive term should get a higher number. The organizers built three datasets, all crowd-annotated on CrowdFlower using Best–Worst Scaling:

  • General English Sentiment Modifiers: a word combined with negators, modals or auxiliaries, and degree adverbs.
  • English Twitter Mixed Polarity: phrases whose parts pull in opposite directions.
  • Arabic Twitter: a word combined with a negator.

Each dataset contains the phrases and also their individual component words. This makes it a "sentiment composition lexicon" that supports intrinsic evaluation of automatic lexicons, which are usually judged only indirectly through sentence classification. Participants received no training data, only a small development set. Five teams sent nine runs. Supervised systems built on existing lexicons did best. They reached a Kendall τ of about 0.7 on general English but stayed far lower on Twitter phrases and Arabic phrases. Composition clearly remains hard, and the released data gives a benchmark for it.

Key points

  • Dataset sizes (total / test):
    • General English: 2,999 / 2,799 terms (1,330 words, 1,469 phrases).
    • English Twitter Mixed Polarity: 1,269 / 1,069 terms (358 words, 711 phrases).
    • Arabic Twitter: 1,366 / 1,166 terms (1,001 words, 165 phrases).
    • Each task also had a 200-term development set that shared no terms with its test set.
  • Annotation procedure:
    • The organizers generated 2×n 4-tuples for n terms, with occurrences of terms and of term pairs kept roughly balanced.
    • Each tuple got at least eight judgments, and at least 80% of responses matched the majority choice.
    • A term's score is the share of "most positive" picks minus the share of "most negative" picks, rescaled to 0–1.
  • Metrics: Kendall's τ was the official metric, with Spearman's ρ also reported. Team rankings came out identical under both.
  • Best systems:
    • ECNU won both English tasks (τ = 0.704 on General English, 0.523 on Mixed Polarity).
    • iLab-Edinburgh won the Arabic task (τ = 0.536).
    • LSIS was the only fully unsupervised entry. It used PMI against sentiment-labelled corpora and against "poor"/"excellent" in Google results.
  • Arabic phrases: correlations were near zero (τ from −0.046 to 0.078), against about 0.5–0.6 for Arabic single words. The authors blame the absence of any manually built phrase-level lexicons for training.

Interesting ideas

  • I1. Best–Worst Scaling turns a graded judgment into easy forced choices inside small sets, then into reliable real-valued scores by simple counting, avoiding unstable rating scales.
  • I2. The lexicon stores scores for whole phrases next to the scores of their component words, so composition rules can be checked directly against human judgments.
  • I3. Evaluation rewards correct ordering (rank correlation) rather than exact numbers, and reports single words and phrases separately so a weakness in composition cannot hide behind easy lexical items.
  • I4. Mixed-polarity phrases (happy tears vs. great loss) show that part-of-speech tags plus word polarities do not determine phrase sentiment; the meaning of the combination matters.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could be asked best/worst questions over 4-tuples to produce graded silver values (e.g., intensity of verbs or adjectives) instead of noisy absolute ratings.
  • I2 (world): the compiled values layer could hold such paired entries and test explicit rules over the UD tree (negator, modal, degree adverb scopes) for how a modifier shifts a head word's value.
  • I3 (learn): evaluation reports could split scores into lexical vs. compositional items and use rank metrics wherever gold values are only meaningful relatively.
  • I4 (frames): pragmatics could treat such pairs as hard cases for evaluative stance, and the dataset could serve as a check that frame-based reasoning beats naive polarity arithmetic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlbertaEdmonton · Canada44 works · 3,236 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Svetlana Kiritchenko, Saif Mohammad and Mohammad Salameh. 2016. SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1004

@inproceedings{kiritchenko2016semeval,
  title = {{SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases}},
  author = {Svetlana Kiritchenko, Saif Mohammad and Mohammad Salameh},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1004},
  url = {https://doi.org/10.18653/v1/s16-1004}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2014. SANA: A large scale multi-genre, multi-dialect lexicon for Arabic subjectivity and sentiment analysis. In Proceedings of the Language Resources and Evaluation Conference (LREC).
  2. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), pages 587–591.
  3. Steven H. Cohen. 2003. Maximum difference scaling: Improved measures of importance and preference for segmentation. Sawtooth Software, Inc. find
  4. Peter Sheridan Dodds, Kameron Decker Harris, Isabel M. Kloumann, Catherine A. Bliss, and Christopher M. Danforth. 2011. Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. PloS One, 6(12):e26752. ↗
  5. Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, et al. 2015. Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences, 112(8):2389–2394. ↗
  6. Samhaa R. El-Beltagy. 2016a. NileTMGR at SemEval-2016 Task 7: Deriving prior polarities for Arabic sentiment terms. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find
  7. Samhaa R. El-Beltagy. 2016b. NileULex: A phrase and word level sentiment lexicon for Egyptian and Modern Standard Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC).
  8. Ramy Eskander and Owen Rambow. 2015. SLSA: A sentiment lexicon for Standard Arabic. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2545– 2550, Lisbon, Portugal.
  9. Andrea Esuli and Fabrizio Sebastiani. 2006. SENTI-WORDNET: A publicly available lexical resource for opinion mining. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC), pages 417–422. find
  10. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, pages 178–201. Edward Elgar Publishing. ↗
  11. Marcus Hartner. 2013. The lingering after-effects in the reader’s mind – an investigation into the affective dimension of literary reading. Journal of Literary Theory Online. find
  12. Amal Htait, Sebastien Fournier, and Patrice Bellot. 2016. LSIS at SemEval-2016 Task 7: Using web search engines for English and Arabic unsupervised sentiment intensity prediction. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
  13. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 168– 177, New York, NY, USA. find
  14. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, pages 81–93. ↗
  15. Svetlana Kiritchenko and Saif M. Mohammad. 2016a. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, California.
  16. Svetlana Kiritchenko and Saif M. Mohammad. 2016b. The effect of negators, modals, and degree adverbs on sentiment composition. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA). find
  17. Svetlana Kiritchenko and Saif M. Mohammad. 2016c. Happy accident: A sentiment composition lexicon for opposing polarity phrases. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find
  18. Svetlana Kiritchenko and Saif M. Mohammad. 2016d. Sentiment composition of words with opposing polarities. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, California. find
  19. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. ↗
  20. Jochen Kleres. 2011. Emotions and narrative analysis: A methodological approach. Journal for the Theory of Social Behaviour, 41(2):182–202. ↗
  21. Ladislav Lenc, Pavel Krl, and Vclav Rajtmajer. 2016. UWB at SemEval-2016 Task 7 : Novel method for automatic sentiment intensity determination. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find
  22. Jordan J. Louviere and George G. Woodworth. 1990. Best-worst analysis. Working Paper. Department of Marketing and Economic Analysis, University of Alberta. find
  23. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  24. Atsunori Minamikawa and Hiroyuki Yokoyama. 2011. Personality estimation based on weblog text classification. In Modern Approaches in Applied Intelligence, pages 89–97. Springer. find
  25. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301–326. ↗
  26. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  27. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation (SemEval), Atlanta, Georgia, USA, June. ↗
  28. Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2015. How translation alters sentiment. Journal of Artificial Intelligence Research, 55:95–130. ↗
  29. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  30. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2016b. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, Submitted. ↗
  31. Saif M Mohammad. 2012. From once upon a time to happily ever after: Tracking emotions in mail and books. Decision Support Systems, 53(4):730–741. ↗
  32. Finn Årup Nielsen. 2011. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC-2011 Workshop on ’Making Sense of Microposts’: Big things come in small packages, pages 93–98. ↗
  33. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. find
  34. Charles E Osgood, George J Suci, and Percy Tannenbaum. 1957. The measurement of meaning. University of Illinois Press. find
  35. Maria Pontiki, Harris Papageorgiou, Dimitrios Galanis, Ion Androutsopoulos, John Pavlopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval), Dublin, Ireland.
  36. Eshrag Refaee and Verena Rieser. 2014. An Arabic Twitter corpus for subjectivity and sentiment analysis. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC).
  37. Eshrag Refaee and Verena Rieser. 2016. iLab-Edinburgh at SemEval-2016 Task 7: A hybrid approach for determining sentiment intensity of Arabic Twitter phrases. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
  38. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment analysis in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval), pages 450– 462, Denver, Colorado.
  39. Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. ↗
  40. Josef Steinberger, Mohamed Ebrahim, Maud Ehrmann, Ali Hurriyetoglu, Mijail Kabadjov, Polina Lenkova, Ralf Steinberger, Hristo Tanev, Silvia Vázquez, and Vanni Zavarella. 2012. Creating sentiment dictionaries via triangulation. Decision Support Systems, 53(4):689–694. Philip Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. ↗
  41. Feixiang Wang, Zhihua Zhang, and Man Lan. 2016. ECNU at SemEval-2016 Task 7: An enhanced supervised learning method for lexicon sentiment intensity ranking. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find
  42. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347–354, Stroudsburg, PA, USA.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kiritchenko-2016-semeval-2016-task-7-determining/