Elamur Research⌕
☰
home / papers / 2017 / SiTAKA at SemEval-2017 Task 4: Sentiment Analysis in Twitter Based on a Rich Set of Features

SiTAKA at SemEval-2017 Task 4: Sentiment Analysis in Twitter Based on a Rich Set of Features

Paper · 2017 · Mohammed Jabreel and Antonio Moreno

Summary

The paper presents SiTAKA, a system built for SemEval-2017 Task 4, subtask A. The task is to label each tweet as positive, negative or neutral, in English and in Arabic. Tweets are hard to classify because they are short, informal and full of misspellings and abbreviations. The authors take a classical feature-engineering approach and feed many hand-designed features into a linear Support Vector Machine. The features are:

  • word n-grams, plus a separate bag of n-grams taken only from negated spans (BonW);
  • POS-tag counts and word/tag bigrams;
  • seven summary scores per sentiment lexicon;
  • Brown and word2vec cluster memberships (English only);
  • pooled pre-trained word embeddings.

Preprocessing lowercases the text, collapses elongated words and marks negation scope with a suffix. Stopwords are deliberately kept. The English model was trained on the pooled SemEval 2013–2016 data. SiTAKA placed 8th of 38 teams in English and 2nd of 8 in Arabic. The result shows that a carefully built feature set with a simple classifier can still compete with neural systems.

Key points

  • Classifier and lexicons: linear-kernel SVM with C = 0.5, tuned on development sets. English used five lexicons (General Inquirer, Hu-Liu, NRC hashtags, SenticNet, TS-Lex). Arabic used four Mohammad & Kiritchenko lexicons, two of which were machine-translated from English.
  • Embeddings: Google News word2vec (300d) and SSWEu (50d) for English, Arabic-SKIP-G300 for Arabic. Each tweet vector concatenates max, min, sum and std pooling over its word vectors.
  • Clusters (English only): 1000 Brown clusters from Ark Tweet NLP (built from 56M tweets) and 4960 k-means clusters over word2vec vectors trained on Twitter data. Tokenization and tagging used Ark Tweet NLP for English and Stanford Tagger for Arabic.
  • Data:
    • English: 27,700 training tweets, 2,684 development tweets, 12,284 test tweets.
    • Arabic: 20,632 training tweets, 671 development tweets, 6,100 test tweets.
  • Official scores:
    • English: macro-recall 0.645, F1^PN 0.628, accuracy 0.643.
    • Arabic: macro-recall 0.550, F1^PN 0.571, accuracy 0.563.

Interesting ideas

  • I1. Negation scope is approximated as the stretch from a negation cue to the next punctuation mark; every word inside gets a suffix, and n-grams from that stretch form their own feature group.
  • I2. Cue words that are ambiguous were removed from the negation list, as with an Arabic word that can also mean "what" or "that", trading recall for fewer false negations.
  • I3. Besides overall lexicon polarity, the system records the score of the last positive and last negative term, on the intuition that sentiment late in a message often decides its overall tone.
  • I4. Stopwords were deliberately kept because earlier work found they carry sentiment signal.
  • I5. Results are reported next to trivial baselines that label every tweet positive, negative or neutral, which exposes how misleading accuracy can be on skewed classes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): instead of a punctuation window, Mova can take negation scope from the neg/advmod subtree in the UD tree and flip the polarity of the frame it governs, keeping the scope decision explainable.
  • I2 (tokens): the tagger and lemma layer could flag cue words with a non-negating reading so that only tokens confirmed by context trigger negation downstream.
  • I3 (frames): when a text holds mixed evaluations (e.g. "good start, but awful ending"), the speech-act/implicature layer could give extra weight to the final evaluative clause, especially after contrastive connectives.
  • I4 (tokens): function words should never be pruned before parsing; this paper is extra evidence that negators and modal particles matter for meaning.
  • I5 (learn): every evaluation of a reader or classifier should include such constant-answer controls, so that a score is shown to beat the majority-class floor rather than just sounding high.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Rovira i VirgiliTarragona · Spain4 works · 220 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Mohammed Jabreel and Antonio Moreno. 2017. SiTAKA at SemEval-2017 Task 4: Sentiment Analysis in Twitter Based on a Rich Set of Features. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2115

@inproceedings{jabreel2017sitaka,
  title = {{SiTAKA at SemEval-2017 Task 4: Sentiment Analysis in Twitter Based on a Rich Set of Features}},
  author = {Mohammed Jabreel and Antonio Moreno},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2115},
  url = {https://doi.org/10.18653/v1/s17-2115}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basant Agarwal, Natasha Mittal, and Erik Cambria. 2013. Enhancing sentiment classification performance using bi-tagged phrases. In Data Mining Workshops (ICDMW), 2013 IEEE 13th International Conference on. IEEE, pages 892–895. find
  2. Erik Cambria, Daniel Olsher, and Dheeraj Rajagopal. 2014. SenticNet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis. In Twenty-eighth AAAI conference on artificial intelligence. find
  3. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119. ↗
  4. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12:2461–2505.
  5. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the State-ofthe-Art in Sentiment Analysis of Tweets. In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013). Atlanta, Georgia, USA. find
  6. Corinna Cortes and Vladimir Vapnik. 1995. Support vector machine. Machine learning 20(3):273–297. find
  7. Li Dong, Furu Wei, Yichun Yin, Ming Zhou, and Ke Xu. 2015. Splusplus: A Feature-Rich Two-stage Classifier for Sentiment Analysis of Tweets. SemEval-2015 page 515. ↗
  8. Alexander Pak and Patrick Paroubek. 2010. Twitter as a Corpus for Sentiment Analysis and Opinion Mining. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, Malta.
  9. Ronen Feldman. 2013. Techniques and applications for sentiment analysis. Communications of the ACM 56(4):82–89. ↗
  10. Bo Pang and Lillian Lee. 2008. Opinion Mining and Sentiment Analysis. Found. Trends Inf. Retr. 2(1-2):1–135. ↗
  11. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs Up?: Sentiment Classification Using Machine Learning Techniques. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10. Association for Computational Linguistics, Stroudsburg, PA, USA, EMNLP ’02, pages 79–86. ↗
  12. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2. Association for Computational Linguistics, Stroudsburg, PA, USA, HLT ’11, pages 42–47. ↗
  13. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
  14. Spence Green and Christopher D Manning. 2010. Better arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, pages 394–402.
  15. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. Semeval-2014 task 9: Sentiment analysis in twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). pages 73–80.
  16. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pages 168– 177. ↗
  17. Hassan Saif, Miriam Fernández, Yulan He, and Harith Alani. 2014. On stopwords, filtering and data sparsity for sentiment analysis of Twitter. In Proceedings of the 9th language resources and evaluation conference (LREC). Reykjavik, Iceland. find
  18. Mohammed Jabreel and Antonio Moreno. 2016. Sen-tirich: Sentiment analysis of tweets based on a rich set of features. In Artificial Intelligence Research and Development - Proceedings of the 19th International Conference of the Catalan Association for Artificial Intelligence, Barcelona, Catalonia, Spain, October 19-21, 2016. pages 137–146. find
  19. Mohammad Salameh Saif M. Mohammad and Svetlana Kiritchenko. 2016. Sentiment lexicons for arabic social media. In Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC). Portorož, Slovenia.
  20. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies 5(1):1–167. ↗
  21. Aliaksei Severyn and Alessandro Moschitti. 2015. UNITN: Training deep convolutional neural network for Twitter sentiment classification. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Association for Computational Linguistics, Denver, Colorado. pages 464–469. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 . ↗
  23. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014c. Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 1555–1565. http://www.aclweb.org/anthology/P14-1146. ↗
  24. Philip Stone, Dexter C Dunphy, Marshall S Smith, and DM Ogilvie. 1968. The general inquirer: A computer approach to content analysis. Journal of Regional Science 8(1):113–116. find
  25. Duyu Tang, Furu Wei, Bing Qin, Ting Liu, and Ming Zhou. 2014a. Coooolll: A Deep Learning System for Twitter Sentiment Classification. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, pages 208–212. http://www.aclweb.org/anthology/S14-2033. ↗
  26. Mohamed A Zahran, Ahmed Magooda, Ashraf Y Mahgoub, Hazem Raafat, Mohsen Rashwan, and Amir Atyia. 2015. Word representations in vector space and their applications for arabic. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pages 430–443. find
  27. Duyu Tang, Furu Wei, Bing Qin, Ming Zhou, and Ting Liu. 2014b. Building Large-Scale Twitter-Specific Sentiment Lexicon : A Representation Learning Approach. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pages 172–182. http://www.aclweb.org/anthology/C14-1018. ↗
  28. Zhixin Zhou, Xiuzhen Zhang, and Mark Sanderson. 2014. Sentiment analysis on twitter through topic-based lexicon expansion. In Databases Theory and Applications, Springer, pages 98–109. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jabreel-2017-sitaka-semeval-2017-task-4/