Elamur Research⌕
☰
home / papers / 2017 / OMAM at SemEval-2017 Task 4: Evaluation of English State-of-the-Art Sentiment Analysis Models for Arabic and a New Topic-based Model

OMAM at SemEval-2017 Task 4: Evaluation of English State-of-the-Art Sentiment Analysis Models for Arabic and a New Topic-based Model

Paper · 2017 · Ramy Baly, Gilbert Badaro, Ali Hamdi et al.

Summary

The paper tackles sentiment analysis of Arabic tweets within SemEval-2017 Task 4. Rich morphology, dialect variation, misspellings and Twitter-specific tokens make this task harder than in English. The OMAM team built four systems:

  • a port of the feature set from the SemEval-2016 English winner, using lemma and character n-grams, punctuation and negation counts, emoticons, POS counts and three Arabic sentiment lexicons;
  • per-cluster classifiers over a k-means–clustered skip-gram embedding space;
  • Recursive Auto Encoders run on morphologically tokenized, parsed text;
  • a new topic-based cascade that guesses a tweet's topic, or a broader domain, and then applies a classifier trained for that topic or domain.

The ported English features worked reasonably well without any Arabic-specific tuning and placed mid-field on message polarity (subtask A). The RAE system lagged, which the authors attribute to MSA-trained tools breaking on noisy tweets. On the topic subtasks, the topic-to-domain backoff came first in subtask C, and a plain topic-agnostic classifier came first in E. The study suggests that transferring English feature engineering is a cheap, strong baseline. It also shows that topic conditioning helps only in some settings.

Key points

  • Subtask A test scores for System 1 were Avg-R 0.438, Avg-F1 0.422 and accuracy 0.430, ranking 5th of 8. On development data, cluster-based System 2 had higher accuracy (0.477) but lower recall.
  • Three lexicons supplied the sentiment features: ArSenL, AraSenti and ADHL. The RAE pipeline used MADAMIRA v2.1 for ATB-scheme tokenization and the Stanford parser. Its trees were left-factored into CNF so that recursive composition would work.
  • The training data covered 34 topics, which the authors mapped onto 8 generic domains. The cascade modules were M1 (unsupervised topic matching), M2 (supervised topic classifier), M3 (domain classifier) and M4 (direct, topic-agnostic classifier).
  • The [M1→M2→M3] cascade ranked 1st in subtask C, with test MAE^M 0.943 and MAE^µ 0.646. Direct M4 ranked 1st in E, and System 1 features ranked 2nd in D (test KLD 0.202). Some of these subtasks had only 2–3 participating teams.
  • In subtask B, ignoring topic information gave the best results. Even so, backing off from topics to domains beat working at the topic level alone.

Interesting ideas

  • I1. A specialist model is tried first, then a coarser-grained model, then a generic fallback whenever the finer level lacks a trained classifier; this lets the system cope with unseen topics.
  • I2. Negation scope is approximated as the span from a negation particle up to the next punctuation mark, giving a cheap but useful signal.
  • I3. The authors blame the weak deep model on parsers trained on clean MSA meeting noisy tweets, which turns genre mismatch in upstream tools into a diagnosable error source.
  • I4. Non-binary parse trees are converted to Chomsky Normal Form by left-factoring, so that a strictly binary composition model can consume them.
  • I5. Lemma n-grams are used instead of surface words to fight sparsity in a morphologically rich language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Ontology lookups for an unknown entity could fall back the same way, from a specific Wikidata class to a WordNet supersense to a generic default. Each fallback step would be logged as part of the explanation.
  • I2 (frames): This works as a fallback heuristic when a UD tree is missing or unreliable. It can also serve as a sanity check against tree-derived negation scope over a frame's arguments.
  • I3 (learn): Evaluations should report parser accuracy separately for each genre, such as fables, word problems and noisy web text. That way, downstream failures can be traced to the tree stage.
  • I4 (tree): A deterministic, reversible binarization of UD trees could feed any binary compositional component while keeping the original arcs recoverable for explanation.
  • I5 (tokens): Matching lexicon and frame triggers on lemmas from crate en, rather than surface forms, would improve coverage of rare inflected forms without losing traceability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

Also on the byline: University of Technology Malaysia.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Ramy Baly, Gilbert Badaro, Ali Hamdi, Rawan Moukalled, Rita Aoun, Georges El-Khoury, Ahmad Al Sallab, Hazem Hajj, Nizar Habash, Khaled Shaban and Wassim El-Hajj. 2017. OMAM at SemEval-2017 Task 4: Evaluation of English State-of-the-Art Sentiment Analysis Models for Arabic and a New Topic-based Model. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2099

@inproceedings{baly2017omam,
  title = {{OMAM at SemEval-2017 Task 4: Evaluation of English State-of-the-Art Sentiment Analysis Models for Arabic and a New Topic-based Model}},
  author = {Ramy Baly, Gilbert Badaro, Ali Hamdi, Rawan Moukalled, Rita Aoun, Georges El-Khoury, Ahmad {Al Sallab}, Hazem Hajj, Nizar Habash, Khaled Shaban and Wassim El-Hajj},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2099},
  url = {https://doi.org/10.18653/v1/s17-2099}
}

References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramy Baly, Roula Hobeica, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, and Ahmad Al-Sallab. 2016. A meta-framework for modeling the human reading process in sentiment analysis. ACM Transactions on Information Systems (TOIS) 35(1):7. ↗
  2. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS) 26(3):12. find
  3. RM Duwairi, Raed Marji, Narmeen Sha’ban, and Sally Rushaidat. 2014. Sentiment analysis in arabic tweets. In Information and communication systems (icics), 2014 5th international conference on. IEEE, pages 1–6. find
  4. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. Samar: Subjectivity and sentiment analysis for arabic social media. Computer Speech & Language 28(1):20–37. ↗
  5. Rasheed M Elawady, Sherif Barakat, and Nora M Elrashidy. 2014. Different feature selection for sentiment classification. International Journal of Information Science and Intelligent System 3(1):137– 150. find
  6. Muhammad Abdul-Mageed, Mona T Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, pages 587– 591.
  7. Noura Farra, Elie Challita, Rawad Abou Assi, and Hazem Hajj. 2010. Sentence-level and document-level sentiment mining for arabic texts. In Data Mining Workshops (ICDMW), 2010 IEEE International Conference on. IEEE, pages 1114–1119. find
  8. Ahmad A Al Sallab, Ramy Baly, Gilbert Badaro, Hazem Hajj, Wassim El Hajj, and Khaled B Shaban. 2015. Deep learning models for sentiment analysis in arabic. In ANLP Workshop 2015. page 9. ↗
  9. Spence Green and Christopher D Manning. 2010. Better arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, pages 394–402.
  10. Ahmad A Al Sallab, Ramy Baly, Gilbert Badaro, Hazem Hajj, Wassim El Hajj, and Khaled B Shaban. in press 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing . find
  11. Nizar Habash and Fatiha Sadat. 2006. Arabic preprocessing schemes for statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers. Association for Computational Linguistics, pages 49–52.
  12. Nora Al-Twairesh, Hend Al-Khalifa, and AbdulMalik Al-Salman. 2016. Arasenti: Large-scale twitter-specific arabic sentiment lexicons. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics pages 697–705.
  13. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies 3(1):1–187. ↗
  14. Mohamed A Aly and Amir F Atiya. 2013. Labr: A large scale arabic book reviews dataset. In ACL (2). pages 494–498. ↗
  15. Ali Hamdi, Khaled Shaban, and Zainal Anazida. 2016. A review on challenging issues in arabic sentiment analysis. Journal of Computer Science 12(9):471– 481. ↗
  16. Gilbert Badaro, Ramy Baly, Rana Akel, Linda Fayad, Jeffrey Khairallah, Hazem Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2015. A light lexicon-based mobile application for sentiment mining of arabic tweets. In ANLP Workshop 2015. page 18. ↗
  17. Roula Hobeica, Hazem Hajj, and Wassim El Hajj. 2011. Machine reading for notion-based sentiment mining. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on. IEEE, pages 75–80. find
  18. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale arabic sentiment lexicon for arabic opinion mining. ANLP 2014 165.
  19. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285 . ↗
  20. Georgios Balikas and Massih-Reza Amini. 2016. Twise at semeval-2016 task 4: Twitter sentiment classification. arXiv preprint arXiv:1606.04351 . ↗
  21. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Bashir Shaban. 2017. A characterization study of arabic twitter data with a benchmarking for state-of-the-art opinion mining models. ANLP 2017 .
  22. Mohamed Maamouri, Dave Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Standard arabic morphological analyzer (sama) version 3.1. Linguistic Data Consortium, Catalog No.: LDC2010L01 . with limited resources. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme. page 16. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119. ↗
  24. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
  25. Miniwatts. 2016. Internet world users by language. http://www.internetworldstats.com/stats7.htm. ↗
  26. Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. J. Artif. Intell. Res.(JAIR) 55:95–130. ↗
  27. Mohammed Rushdi-Saleh, M Teresa Martı́n-Valdivia, L Alfonso Ureña-López, and José M Perea-Ortega. 2011. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology 62(10):2045–2054. ↗
  28. Asmaa Mountassir, Houda Benbrahim, and Ilham Berrada. 2012. An empirical study to address the problem of unbalanced data sets in sentiment classification. In Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference on. IEEE, pages 3298–3303. find
  29. Amira Shoukry and Ahmed Rafea. 2012. Sentence-level arabic sentiment analysis. In Collaboration Technologies and Systems (CTS), 2012 International Conference on. IEEE, pages 546–550. find
  30. Mahmoud Nabil, Mohamed A Aly, and Amir F Atiya. 2015. Astd: Arabic sentiment tweets dataset. In EMNLP. pages 2515–2519. ↗
  31. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 151–161.
  32. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PloS one 10(12):e0144296. ↗
  33. Nazlia Omar, Mohammed Albared, Adel Qasem Al-Shabi, and Tareq Al-Moslmi. 2013. Ensemble of classification algorithms for subjectivity and sentiment analysis of arabic customers’ reviews. International Journal of Advancements in Computing Technology 5(14):77. find
  34. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP). Citeseer, volume 1631, page 1642.
  35. Chukwuyem J. Onyibe and Nizar Habash. 2017. OMAM at SemEval-2017 task 4: English sentiment analysis with conditional random fields. In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
  36. Kazem Taghva, Rania Elkhoury, and Jeffrey Coombs. 2005. Arabic stemming without a root dictionary. In Information Technology: Coding and Computing, 2005. ITCC 2005. International Conference on. IEEE, volume 1, pages 152–157. find
  37. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC. volume 14, pages 1094–1101.
  38. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP. pages 1422– 1432.
  39. Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 417–424.
  40. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12(Oct):2825–2830. ↗
  41. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework. In LREC. pages 2362–2369. ↗
  42. Eshrag Refaee and Verena Rieser. 2014a. Can we read emotions from a smiley face? emoticon-based distant supervision for subjectivity and sentiment analysis of arabic twitter feeds. In 5th International Workshop on Emotion, Social Signals, Sentiment and Linked Open Data, LREC. find
  43. Eshrag Refaee and Verena Rieser. 2014b. Subjectivity and sentiment analysis of arabic twitter feeds find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baly-2017-omam-semeval-2017-task-4/