Elamur Research⌕
☰
home / papers / 2013 / SemEval-2013 Task 2: Sentiment Analysis in Twitter

SemEval-2013 Task 2: Sentiment Analysis in Twitter

Summary

Research on sentiment in social media had no shared, openly available benchmark. Existing Twitter sentiment sets were small and proprietary, limited to Spanish, or labelled noisily through emoticons and hashtags. None annotated sentiment at the level of individual expressions. The organisers built a SemEval shared task with two subtasks. Subtask A asks for the polarity of a marked phrase in its context. Subtask B asks for the overall polarity of a whole message. Tweets about trending named entities were collected over a year and pre-filtered with SentiWordNet. Five Amazon Mechanical Turk workers then annotated each message at both phrase and message level. A test set of SMS messages from the NUS SMS corpus was added to check cross-genre transfer. Systems were scored with F1 averaged over the positive and negative classes. The task drew 149 runs from 44 teams. The best scores were about 88.9 F1 on subtask A and 69 F1 on subtask B, so whole-message polarity proved considerably harder. The released corpus became a common reference for Twitter sentiment work.

Key points

  • Data collection:
    • Tweets were gathered via the streaming API between January 2012 and January 2013.
    • Topics were found by running a Twitter-tuned NER system and keeping entities that spike around specific dates.
    • Test tweets cover later periods and different topics than the training tweets.
  • Annotation:
    • Each sentence had 5 Turkers, each paid 3 cents per HIT.
    • Workers needed an approval rate above 95% and at least 50 approved HITs.
    • For subtask A, a word counted as subjective, or received a polarity, only when 2/3 of the annotations agreed.
    • For subtask B, the majority label was used, and ties were discarded.
  • Pre-filtering: a message was kept only if it contained at least one word whose SentiWordNet positive or negative score exceeded 0.3 for some sense. This was done to limit class imbalance.
  • Leading systems:
    • NRC-Canada ranked first in three of the four conditions: A-Twitter 88.93, B-Twitter 69.02 and B-SMS 68.46.
    • GU-MLT-LT topped A-SMS with 88.37.
    • The best system in every condition was a constrained one.
  • Common ingredients across teams:
    • Classifiers: SVM, MaxEnt and Naive Bayes.
    • Lexicons: MPQA, SentiWordNet, AFINN and Bing Liu's Opinion Lexicon.
    • Heavy Twitter-specific preprocessing.
    • The data was released under CC BY 3.0.

Interesting ideas

  • I1. Defining "constrained" by training data only. Using an external lexicon as a feature source still counts as constrained. Using that lexicon to auto-label extra training messages does not. This draws a clean, checkable line that other shared tasks could adopt.
  • I2. Estimating a ceiling from annotator agreement. Lower, average and upper bounds were computed per HIT and then averaged. This shows how close systems can get to human consistency before anyone overinterprets small score gaps.
  • I3. Rule-based spam filtering of crowd work. Responses were rejected if they had overlapping spans, a "subjective" label with no phrase, every word marked, or no overall label. These are cheap structural checks that transfer to any span-annotation crowdsourcing project, such as a Ukrainian corpus.
  • I4. Collapsing categories annotators confuse. Merging "objective" and "neutral" reduced the number of discarded items, a pragmatic trade-off between label granularity and data yield.
  • I5. Testing on an unseen genre and period. SMS was used only for testing, and test tweets came from later times and new topics. This checks robustness against topic and domain shift rather than memorisation, which suits any low-resource setting where in-genre training data is scarce.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

200 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 188 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nakov2013semeval,
  title = {{SemEval-2013 Task 2: Sentiment Analysis in Twitter}},
  author = {Nakov, Preslav and Rosenthal, Sara and Kozareva, Zornitsa and Stoyanov, Veselin and Ritter, Alan and Wilson, Theresa},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)},
  year = {2013},
  url = {https://aclanthology.org/S13-2052/}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Nicoletta Calzolari (chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias, ed… find our card
  2. Luciano Barbosa and Junlan Feng. 2010. Robust sentiment detection on Twitter from biased and noisy data. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, COLING ’10, pages 36–44, Beijing, China. find our card
  3. Albert Bifet, Geoffrey Holmes, Bernhard Pfahringer, and Ricard Gavaldà. 2011. Detecting sentiment change in Twitter streaming data. Journal of Machine Learning Research - Proceedings Track, 17:5–11. find
  4. Tao Chen and Min-Yen Kan. 2012. Creating a live, public short message service corpus: the NUS SMS corpus. Language Resources and Evaluation, pages 1– 37. find
  5. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Uppsala, Sweden. find our card
  6. Bernard J. Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury. 2009. Twitter power: Tweets as electronic word of mouth. J. Am. Soc. Inf. Sci. Technol., 60(11):2169–2188. find
  7. Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The Good the Bad and the OMG! In Lada A. Adamic, Ricardo A. Baeza-Yates, and Scott Counts, editors, Proceedings of the Fifth International Conference on Weblogs and Social Media, ICWSM’ 11, pages 538–541, Barc… find
  8. Brendan O’Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010. From tweets to polls: Linking text sentiment to public opinion time series. In William W. Cohen and Samuel Gosling, editors, Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM ’… find
  9. Alexander Pak and Patrick Paroubek. 2010. Twitter based system: Using Twitter for disambiguating sentiment ambiguous adjectives. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pages 436–439, Los Angeles, CA, USA. find
  10. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 1524–1534, Edinburgh, United Kingdom. find our card
  11. Alan Ritter, Mausam, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’12, pages 1104–1112, Beijing, China. find
  12. Andranik Tumasjan, Timm O. Sprenger, Philipp G. Sandner, and Isabell M. Welpe. 2010. Predicting elections with Twitter: What 140 characters reveal about political sentiment. In William W. Cohen and Samuel Gosling, editors, Proceedings of the Fourth International Conference on Weblogs and Social Medi… find
  13. Julio Villena-Román, Sara Lana-Serrano, Eugenio Martı́nez-Cámara, and José Carlos González Cristóbal. 2013. TASS - Workshop on Sentiment Analysis at SEPLN. Procesamiento del Lenguaje Natural, 50:37–44. find
  14. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165–210. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakov-2013-semeval-2013-task-2-sentiment/