Elamur Research⌕
☰
home / papers / 2014 / SemEval-2014 Task 9: Sentiment Analysis in Twitter

SemEval-2014 Task 9: Sentiment Analysis in Twitter

Summary

This paper reports on the second edition of the SemEval shared task on Twitter sentiment, which follows SemEval-2013 Task 2. The task has two subtasks. Subtask A asks for the polarity of a marked word or phrase in its context. Subtask B asks for the polarity of a whole message. Labels are positive, negative or neutral in both. The organizers reused the 2013 Twitter and SMS data and added three new test sets: regular tweets, tweets carrying #sarcasm, and LiveJournal sentences. All five test sets were mixed into one shuffled file, so the hidden sets also measured how well systems transfer to new genres. Systems were scored by averaging the F1 of the positive and negative classes. It was again the most popular SemEval task, with 46 teams. On the official Twitter2014 set, NRC-Canada won subtask A and TeamX won subtask B. Sarcastic tweets proved clearly harder for subtask B systems. The task gives the community a shared, extensible benchmark for informal social-media sentiment.

Key points

  • Participation: subtask A had 27 submissions from 21 teams, and subtask B had 50 submissions from 44 teams. Best Twitter2014 scores were 86.63 (NRC-Canada, A) and 70.96 (TeamX, B).
  • Test sets were Twitter2014-test, Twitter2014-sarcasm (small: 86 messages for B), LiveJournal2014-test, and the 2013 progress sets Twitter2013-test and SMS2013-test. The SMS data comes from the NUS SMS corpus.
  • Tweets were released only as IDs with a download script, to respect Twitter's terms of service. As a result, teams obtained between 5,215 and 10,882 training tweets. The top two subtask B teams used fewer than 8,500, so the gap mattered little.
  • SVM, MaxEnt and Naive Bayes were the dominant classifiers, combined with heavy use of sentiment lexicons. Two deep-learning systems took part: coooolll placed 2nd and ThinkPositive placed 10th in subtask B.
  • Few systems were unconstrained (7 of 27 in A, 8 of 50 in B). The winners were constrained, and extra training data did not reliably help.

Interesting ideas

  • I1. Hidden out-of-domain test sets. New-genre and progress test sets were merged and shuffled into the official test file without revealing their sources. This yields an honest generalization check without extra work from participants, and it is a cheap way to probe domain robustness in any benchmark.
  • I2. Removing the label cue during annotation. Sarcastic tweets were shown to annotators with the #sarcasm hashtag stripped, and annotators also rated how sarcastic each tweet was. This prevents a surface marker from biasing the labels and could be reused for any phenomenon signalled by an explicit tag.
  • I3. Averaging only positive and negative F1. The metric ignores the neutral class, so systems cannot score well just by predicting the frequent "neutral" label. This suits skewed label distributions in other classification tasks.
  • I4. Defining "constrained" by training data only. External lexicons are allowed in constrained systems, but automatically labelling extra messages for training is not. This precise rule makes comparisons fairer and is a useful template for defining tracks in shared tasks.
  • I5. Persistent progress test sets. Each edition is re-evaluated on the earlier year's test data, which tracks progress over time on a fixed benchmark. Keeping it undisclosed in the merged file discourages over-tuning on it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

124 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/s14-2009

@inproceedings{rosenthal2014semeval,
  title = {{SemEval-2014 Task 9: Sentiment Analysis in Twitter}},
  author = {Rosenthal, Sara and Ritter, Alan and Nakov, Preslav and Stoyanov, Veselin},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2009},
  url = {https://doi.org/10.3115/v1/s14-2009}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation, LREC ’10, Valletta, Malta. find our card
  2. Luciano Barbosa and Junlan Feng. 2010. Robust sentiment detection on Twitter from biased and noisy data. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, COLING ’10, pages 36–44, Beijing, China. find our card
  3. Albert Bifet, Geoffrey Holmes, Bernhard Pfahringer, and Ricard Gavaldà. 2011. Detecting sentiment change in Twitter streaming data. Journal of Machine Learning Research, Proceedings Track, 17:5– 11. find
  4. Tao Chen and Min-Yen Kan. 2013. Creating a live, public short message service corpus: the NUS SMS corpus. Language Resources and Evaluation, 47(2):299–335. find
  5. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Uppsala, Sweden. find our card
  6. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Assoc… find our card
  7. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Short Papers, ACL-HLT ’11, pages 581–586, Portland, Oregon,… find our card
  8. Bernard Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury. 2009. Twitter power: Tweets as electronic word of mouth. J. Am. Soc. Inf. Sci. Technol., 60(11):2169–2188. find
  9. Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The good the bad and the OMG! In Proceedings of the Fifth International Conference on Weblogs and Social Media, ICWSM ’11, Barcelona, Catalonia, Spain. find
  10. Christine Liebrecht, Florian Kunneman, and Antal Van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets \#not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37, Atlanta, Georgia, USA. find our card
  11. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-ofthe-art in sentiment analysis of tweets. In Proceedings of the Seventh international workshop on Semantic Evaluation Exercises, SemEval-2013, pages 321–327, Atlanta, Georgia, USA. find
  12. Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 task 2: Sentiment analysis in Twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Sema… find our card
  13. Brendan O’Connor, Ramnath Balasubramanyan, Bryan Routledge, and Noah Smith. 2010. From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM ’10, Washington, DC, USA. find
  14. Alexander Pak and Patrick Paroubek. 2010. Twitter based system: Using Twitter for disambiguating sentiment ambiguous adjectives. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pages 436–439, Uppsala, Sweden. find
  15. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: Sentiment classification using machine learning techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pages 79–86. find our card
  16. Maria Pontiki, Harris Papageorgiou, Dimitrios Galanis, Ion Androutsopoulos, John Pavlopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval ’14, Dublin, Ireland. find our card
  17. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 1524–1534, Edinburgh, Scotland, UK. find our card
  18. Andranik Tumasjan, Timm Sprenger, Philipp Sandner, and Isabell Welpe. 2010. Predicting elections with Twitter: What 140 characters reveal about political sentiment. In Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM ’10, Washington, DC, USA. find
  19. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165–210. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosenthal-2014-semeval-2014-task-9-sentiment/