Elamur Research⌕
☰
home / papers / 2017 / Tw-StAR at SemEval-2017 Task 4: Sentiment Classification of Arabic Tweets

Tw-StAR at SemEval-2017 Task 4: Sentiment Classification of Arabic Tweets

Paper · 2017 · Hala Mulki, Hatem Haddad, Mourad Gridach et al.

Summary

The paper describes the Tw-StAR team's entry to SemEval-2017 subtask 4A-Arabic, which asks systems to label each Arabic tweet as positive, negative or neutral. Arabic tweets are hard to classify because of rich morphology, informal spelling, and several dialects in which the same expression can carry opposite sentiment. The authors clean the tweets by removing URLs, usernames, punctuation and non-Arabic text, and they filter stopwords with a merged MSA, Egyptian, Levantine and Gulf list that deliberately keeps negation words. They then compare two systems. The first is supervised, using binary presence features from unigrams to trigrams with SVM and NB classifiers. The second is lexicon-based: it sums constant ±1 weights over a merged set of pre-built and hand-made dialect lexica. On the DEV set, linear SVM performed best, and it was chosen for the final submission. That run placed 7th of 8 on average F1 for TEST. The lexicon model came close to SVM when stemming was switched off, since the ISRI stemmer mangles dialectal words.

Key points

  • The SemEval data has 3355 labelled tweets, split into 2684 for TRAIN and 671 for DEV, plus 6100 TEST tweets. Manual annotation shows a mostly MSA set: 2643 MSA, 393 Gulf, 247 Egyptian, 69 Levantine and 3 Moroccan.
  • Features are True/False presence flags for the 40,100 most frequent uni-, bi- and trigrams, selected with NLTK FreqDist.
  • Results on DEV:
    • The baseline scored 0.249 F1 and 0.333 recall.
    • SVM (LIBSVM) scored 0.384 F1 and 0.459 recall.
    • NB (Scikit-Learn) scored 0.284 F1 and 0.418 recall.
    • The lexicon model reached 0.342 F1 without stemming and dropped to 0.309 with ISRI stemming.
  • The merged lexica hold 6587 entries in total:
    • NileULex: 5953
    • Arabic Hashtag Lexicon seeds: 230
    • Arabic Emotion Lexicon seeds: 23
    • Manual Levantine lexicon: 281
    • Manual Gulf lexicon: 100
  • Official TEST scores were 0.416 average F1 (rank 7/8), 0.431 average recall (7/8) and 0.454 accuracy (5/8).

Interesting ideas

  • I1. Negation words are removed from the stopword list before filtering, so cleanup cannot erase words that flip polarity; a generic stopword list would silently destroy that signal.
  • I2. Lexicon lookup tries the surface form first and only falls back to the stem when nothing matches, because aggressive stemming can turn a neutral word into a different, polar root.
  • I3. An ablation with and without the stemmer showed that a preprocessing step everyone assumes is helpful actually hurt results on dialectal text.
  • I4. Stopwords are kept during lexicon matching (though removed for n-gram features) so that multi-word lexicon entries can still match in full.
  • I5. Annotating the training set by dialect showed which varieties the lexica must cover, and small hand-built lexica were then added for the under-served Levantine and Gulf varieties.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Audit any normalisation or filtering list in crate en so that negators like "not", "never" and "no" are always protected and reach the tree as tokens.
  • I2 (world): When mapping words to WordNet/VerbNet entries, prefer exact surface or lemma matches and treat stem-level matches as a lower-confidence fallback that is flagged in the explanation.
  • I3 (learn): Add switch-off ablations for each pipeline stage (lemmatizer, normaliser) to evaluation runs, so that stages which harm non-standard input such as informal or dialectal English get caught.
  • I4 (frames): For multiword expressions and idioms with fixed sentiment or speech-act force, match on the unfiltered token sequence so that function words inside the phrase are not lost.
  • I5 (learn): Profile silver and test data by register or variety, and use the profile to decide where small targeted hand-built resources or extra teacher-labelled data are most worth adding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Libre de BruxellesBrussels · Belgium2 works · 25 citations

Also on the byline: Selçuk University · Université Ibn Zohr.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Hala Mulki, Hatem Haddad, Mourad Gridach and Ismail Babaoglu. 2017. Tw-StAR at SemEval-2017 Task 4: Sentiment Classification of Arabic Tweets. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2110

@inproceedings{mulki2017tw,
  title = {{Tw-StAR at SemEval-2017 Task 4: Sentiment Classification of Arabic Tweets}},
  author = {Hala Mulki, Hatem Haddad, Mourad Gridach and Ismail Babaoglu},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2110},
  url = {https://doi.org/10.18653/v1/s17-2110}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samhaa R. El-Beltagy. 2016. Nileulex: A phrase and word level sentiment lexicon for egyptian and modern standard arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA).
  2. Nawaf A Abdulla, Nizar A Ahmed, Mohammed A Shehab, and Mahmoud Al-Ayyoub. 2013. Arabic sentiment analysis: Lexicon-based and corpus-based. In 2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT). pages 1–6. find
  3. Samhaa R El-Beltagy and Ahmed Ali. 2013. Open issues in the sentiment analysis of Arabic social media: A case study. In 9th the International Conference on Innovations and Information Technology (IIT2013). IEEE, pages 215–220. find
  4. Nawaf A Abdulla, Nizar A Ahmed, Mohammed A Shehab, Mahmoud Al-Ayyoub, Mohammed N Al-Kabi, and Saleh Al-rifai. 2016. Towards improving the lexicon-based approach for arabic sentiment analysis. In Big Data: Concepts, Methodologies, Tools, and Applications. IGI Global, pages 1970–1986. ↗
  5. Nagwa El-Makky, Khaled Nagi, Alaa El-Ebshihy, Esraa Apady, Omneya Hafez, Samar Mostafa, and Shimaa Ibrahim. 2014. Sentiment analysis of colloquial Arabic tweets. In ASE Big-Data/SocialInformatics/PASSAT/BioMedCom 2014 Conference, Harvard University. find
  6. Soha Ahmed, Michel Pasquier, and Ghassan Qadah. 2013. Key issues in conducting sentiment analysis on Arabic social media text. In 2013 9th International Conference on Innovations in Information Technology (IIT). IEEE, pages 72–77. find
  7. Lamia Al-Horaibi and Muhammad Badruddin Khan. 2016. Sentiment Analysis of Arabic Tweets Using Semantic Resources. International Journal of Computing & Information Sciences 12(2):149. find
  8. Hady ElSahar and Samhaa R El-Beltagy. 2015. Building large arabic multi-domain resources for sentiment analysis. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pages 23–34. find
  9. Salha Al-Osaimi and Khan Muhammad Badruddin. 2014. Role of Emotion icons in Sentiment classification of Arabic Tweets. In Proceedings of the 6th international conference on management of emergent digital ecosystems. Association for Computing Machinery (ACM), pages 167–171. find
  10. M M Itani, R N Zantout, L Hamandi, and I Elka-bani. 2012. Classifying sentiment in arabic social networks: Naı̈ve search versus Naı̈ve Bayes. In 2012 2nd International Conference on Advances in Computational Tools for Engineering Applications (ACTEA). IEEE, pages 192–197. find
  11. Sarah O Alhumoud, Mawaheb I Altuwaijri, Tarfa M Albuhairi, and Wejdan M Alohaideb. 2015. Survey on Arabic Sentiment Analysis in Twitter. International Science Index 9(1):364–368. ↗
  12. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 142–150.
  13. Mariam Biltawi, Wael Etaiwi, Sara Tedmori, Amjad Hudaib, and Arafat Awajan. 2016. Sentiment classification techniques for Arabic language: A survey. In 2016 7th International Conference on Information and Communication Systems (ICICS). IEEE, pages 339–346. find
  14. Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How Translation Alters Sentiment. J. Artif. Intell. Res.(JAIR) 55:95–130. ↗
  15. Steven Bird. 2006. NLTK: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions. Association for Computational Linguistics, pages 69–72.
  16. Mahmoud Nabil, Mohamed A Aly, and Amir F Atiya. 2015. ASTD: Arabic Sentiment Tweets Dataset. In Empirical Methods on Natural Language Processing (EMNLP). pages 2515–2519. ↗
  17. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2(3):1–27. find
  18. Venkata Sasank Pagolu, Kamal Nayan Reddy Challa, Ganapati Panda, and Babita Majhi. 2016. Sentiment Analysis of Twitter Data for Predicting Stock Market Movements. In 2016 International Conference on Signal Processing, Communication, Power and Embedded System. ↗
  19. Mohamed Y Dahab, Al Ibrahim, and Rihab Al-Mutawa. 2015. A comparative study on arabic stemmers. International Journal of Computer Applications 125(8):38–47. ↗
  20. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and Others. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825–2830. ↗
  21. R M Duwairi, Raed Marji, Narmeen Sha’ban, and Sally Rushaidat. 2014. Sentiment analysis in arabic tweets. In 2014 5th International Conference on Information and Communication Systems (ICICS). IEEE, pages 1–6. find
  22. Stefan Räbigera, Mishal Kazmia, Yücel Saygına, Peter Schüllerb, and Myra Spiliopoulouc. 2016. SteM at SemEval-2016 Task 4: Applying active learning to improve sentiment classification. Proceedings of SemEval pages 64–70. find
  23. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. ACM, pages 97–106. find
  24. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
  25. Mohammed Rushdi-Saleh, M Teresa Martı́n-Valdivia, L Alfonso Ureña-López, and José M Perea-Ortega. 2011. OCA: Opinion corpus for Arabic. Journal of the American Society for Information Science and Technology 62(10):2045–2054. ↗
  26. Hassan Saif, Yulan He, Miriam Fernandez, and Harith Alani. 2016. Contextual semantics for sentiment analysis of Twitter. Information Processing & Management 52(1):5–19. ↗
  27. Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after Translation: A Case-Study on Arabic Social Media Posts. In 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. The Association for Computational Linguistics, pages 767–777.
  28. Amira Shoukry and Ahmed Rafea. 2012. Sentence-level Arabic sentiment analysis. In 2012 International Conference on Collaboration Technologies and Systems (CTS). IEEE, pages 546–550. find
  29. Duyu Tang, Bing Qin, and Ting Liu. 2015. Deep learning for sentiment analysis: Successful approaches and future challenges. Wiley Int. Rev. Data Min. and Knowl. Disc. 5(6):292–303. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mulki-2017-tw-star-semeval-2017-task/