Elamur Research⌕
☰
home / papers / 2018 / EMA at SemEval-2018 Task 1: Emotion Mining for Arabic

EMA at SemEval-2018 Task 1: Emotion Mining for Arabic

Paper · 2018 · Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj et al.

Summary

The paper describes EMA, a system built by a team at the American University of Beirut for the Arabic track of SemEval-2018 Task 1 (Affect in Tweets). Opinion mining for Arabic is well studied, but emotion mining is not. The task had five subtasks: emotion intensity regression and ordinal classification, valence regression and ordinal classification, and multi-label emotion classification. The training sets were small, ranging from about 800 to 2278 tweets. The team compared preprocessing variants, feature types and many classical learners. The best pipeline normalized Arabic characters, removed diacritics and elongation, replaced emojis with Arabic words and applied light stemming. Word embeddings were then fed to task-specific models, usually ensembles of Ridge, SVM and Random Forest. Deep networks did worse, which the authors attribute to the small amount of data. EMA placed first in multi-label emotion classification (subtask 5) and third in both regression subtasks. The work shows that careful normalization and simple regularized ensembles can compete in low-resource, dialect-heavy settings.

Key points

  • Data sizes:
    • Subtasks 1–2: roughly 800 training and 200 dev tweets per emotion (anger, fear, joy, sadness).
    • Subtasks 3–4: 932 training and 138 dev tweets.
    • Subtask 5: 2278 training and 585 dev tweets, with 11 emotion labels plus neutral.
  • Preprocessing:
    • Normalization rules follow Shoukry and Rafea: hamza, t marbouta and ya' maqsoura are normalized, and diacritics are removed.
    • Elongations and non-Arabic letters are dropped.
    • A hand-built lexicon maps 100 frequent emojis to Arabic words.
    • Stemming uses ARLSTEM. Stems beat lemmas, likely because the available analyzers are trained on MSA while the tweets are mostly dialectal.
  • Features tested:
    • n-grams.
    • Emotion lexicon ArSEL and sentiment lexicon ArSenL.
    • Embeddings from AraVec and FastText, which performed best overall.
  • Test results by subtask:
    • Subtask 1: an RR+SVR+RF ensemble reached an average Pearson of 0.643, placing 3rd (best team AffecThor: 0.685).
    • Subtask 2: SVC dropped from 0.507 on dev to 0.267 on test, placing 8th.
    • Subtask 3: the ensemble reached Pearson 0.804, placing 3rd (best: 0.8284).
    • Subtask 4: an RC+SVC ensemble reached 0.643, placing 5th (best EiTAKA: 0.809).
    • Subtask 5: a linear SVC with L1 reached accuracy 0.489, placing 1st.
  • Deep models: CNN and LSTM-based networks underperformed the classical models. The authors blame the very small training sets.

Interesting ideas

  • I1. Emojis are rewritten as ordinary words through a small hand-made lexicon, so their emotional signal travels through the same pipeline as normal vocabulary instead of being discarded as noise.
  • I2. When the morphological analyzer was trained on a different register (MSA) than the input (dialect), a crude light stemmer beat full lemmatization; robustness can matter more than linguistic depth under domain shift.
  • I3. The team ensembled models with complementary biases: Ridge for shrunken linear weights, SVM for margins, and a Random Forest with many trees for nonlinearity. This gave stable gains on tiny datasets where single models and deep nets were unreliable.
  • I4. The paper treats preprocessing and feature choice as shared across tasks but lets the predictor vary by subtask. This keeps one common front end and tunes only the final layer.
  • I5. In subtask 2, SVC looked fine on dev but scored much lower on test. With dev sets of a few hundred items, model selection on dev alone is fragile.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tokenizer could map emojis and emoticons to English lemmas tagged with a marker, so later stages see them as interpretable words with a known origin.
  • I2 (tokens): when lemmatizing informal or nonstandard English, measure whether a lightweight fallback stemmer beats the main analyzer on unknown forms, and use it only there.
  • I3 (learn): small silver-data tasks such as repair-rule scoring could combine a few cheap, differently biased learners rather than one large model, while keeping each member's decision inspectable.
  • I4 (answer): different readers (tests, word problems, fable QA) could share one parsing and frame front end and differ only in a small task-specific decision module.
  • I5 (learn): evaluations on small held-out sets should report variance or use additional held-out slices and negative controls before a model or rule is declared better.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj and Wassim El-Hajj. 2018. EMA at SemEval-2018 Task 1: Emotion Mining for Arabic. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1036

@inproceedings{badaro2018ema,
  title = {{EMA at SemEval-2018 Task 1: Emotion Mining for Arabic}},
  author = {Gilbert Badaro, Obeida {El Jundi}, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj and Wassim El-Hajj},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1036},
  url = {https://doi.org/10.18653/v1/s18-1036}
}

References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014a. A large scale Arabic sentiment lexicon for Arabic opinion mining. ANLP 2014, 165.
  2. Kheireddine Abainia, Siham Ouamour, and Halim Sayoud. 2017. A novel robust arabic light stemmer. Journal of Experimental & Theoretical Artificial Intelligence, 29:557–573. ↗
  3. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, Wassim El-hajj, and Khaled Shaban. 2014b. An efficient model for sentiment classification of Arabic tweets on mobiles. In Qatar Ramy Baly, Gilbert Badaro, Ali Hamdi, Rawan Moukalled, Rita Aoun, Georges El-Khoury, Ahmad Al Sallab, Hazem Hajj, Nizar Habash, Khaled Shaban, et al. 2017b. Omam at semeval-2017 task 4: Evaluation of english state-of-the-art sentiment analysis models for arabic and a new topic-based model. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 603–610. Foundation Annual Research Conference, 1, page ITPP0631.
  4. Gilbert Badaro, Hazem Hajj, Wassim El-Hajj, and Lama Nachman. 2013. A hybrid approach with collaborative filtering for recommender systems. In Wireless Communications and Mobile Computing Conference (IWCMC), 2013 9th International, pages 349–354. IEEE. find
  5. Gilbert Badaro, Hazem Hajj, Ali Haddad, Wassim El-Hajj, and Khaled Bashir Shaban. 2014c. A multiresolution approach to recommender systems. In Proceedings of the 8th Workshop on Social Network Mining and Analysis, page 9. ACM. find
  6. Ramy Baly, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Khaled Bashir Shaban, and Wassim El-Hajj. 2017c. Comparative evaluation of sentiment analysis methods across arabic dialects. Procedia Computer Science, 117:266–273. find
  7. Gilbert Badaro, Hazem Hajj, Ali Haddad, Wassim El-Hajj, and Khaled Bashir Shaban. 2014d. Recommender systems using harmonic analysis. In Data Mining Workshop (ICDMW), 2014 IEEE International Conference on, pages 1004– 1011. IEEE. find
  8. Anil Bandhakavi, Nirmalie Wiratunga, Stewart Massie, and Deepak Padmanabhan. 2017. Lexicon generation for emotion detection from text. IEEE intelligent systems, 32(1):102–108. ↗
  9. Claudia Beleites, Ute Neugebauer, Thomas Bocklitz, Christoph Krafft, and Jürgen Popp. 2013. Sample size planning for classification models. Analytica chimica acta, 760:25–33. ↗
  10. Gilbert Badaro, Hussein Jundi, Hazem Hajj, and Wassim El-Hajj. 2018a. Emowordnet: Automatic expansion of emotion lexicon using english wordnet. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM2018) co-located with NAACL2018.
  11. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
  12. Gilbert Badaro, Hussein Jundi, Hazem Hajj, Wassim El-Hajj, and Nizar Habash. 2018b. Arsel: A large scale arabic sentiment and emotion lexicon. In Proceedings of the 3rd Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT3) co-located with LREC2018. find
  13. Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1–8. ↗
  14. Roger Bougie, Rik Pieters, and Marcel Zeelenberg. 2003. Angry customers don’t come back, they get back: The experience and behavioral implications of anger and dissatisfaction in services. Journal of the Academy of Marketing Science, 31(4):377–393. ↗
  15. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Shaban. 2017a. A characterization study of arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 110–118.
  16. Layale Constantine, Gilbert Badaro, Hazem Hajj, Wassim El-Hajj, Lama Nachman, Mohamed BenSaleh, and Abdulfattah Obeid. 2016. A framework for emotion recognition from human computer interaction in natural setting. 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2016), Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM 2016). find
  17. Ramy Baly, Gilbert Badaro, Hazem Hajj, Nizar Habash, Wassim El Hajj, and Khaled Shaban. 2014. Semantic model representation for human’s pre-conceived notions in arabic text with applications to sentiment mining. In Qatar Foundation Annual Research Conference, 1, page ITPP1075. find
  18. Dipankar Das, Soujanya Poria, and Sivaji Bandyopadhyay. 2012. A classifier based approach to Saif M Mohammad and Peter D Turney. 2010. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, pages 26–34. Association for Computational Linguistics. emotion lexicon construction. In NLDB, pages 320–326. Springer.
  19. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. ↗
  20. Amira F El Gohary, Torky I Sultan, Maha A Hana, and Mohamed M El Dosoky. 2013. A computational approach for analyzing and detecting emotions in arabic text. International Journal of Engineering Research and Applications (IJERA), 3:100–107. find
  21. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  22. Martin Hibbeln, Jeffrey L Jenkins, Christoph Schneider, Joseph S Valacich, and Markus Weinmann. 2017. How is your user feeling? inferring emotion through human–computer interaction devices. MIS Quarterly, 41(1). find
  23. Saif M Mohammad and Tony Wenda Yang. 2011. Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis, pages 70–79. Association for Computational Linguistics. find
  24. Dung T Ho and Tru H Cao. 2012. A high-order hidden markov model for emotion detection from textual data. In Pacific Rim Knowledge Acquisition Workshop, pages 94–105. Springer. find
  25. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1– 135. ↗
  26. Ali Houjeij, Layla Hamieh, Nader Mehdi, and Hazem Hajj. 2012. A novel approach for emotion classification based on fusion of text and speech. In Telecommunications (ICT), 2012 19th International Conference on, pages 1–6. IEEE. find
  27. Amol S Patwardhan and Gerald M Knapp. 2017. Multimodal affect analysis for product feedback assessment. arXiv preprint arXiv:1705.02694. ↗
  28. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. Theories of emotion, 1(3-31):4. ↗
  29. Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In Mining text data, pages 415–463. Springer. ↗
  30. Robert Plutchik. 1994. The psychology and biology of emotion. HarperCollins College Publishers. find
  31. Saif M Mohammad. 2017. Word affect intensities. arXiv preprint arXiv:1704.08798. ↗
  32. Soujanya Poria, Alexander Gelbukh, Dipankar Das, and Sivaji Bandyopadhyay. 2012. Fuzzy clustering for semi-supervised learning–case study: Construction of an emotion lexicon. In Mexican International Conference on Artificial Intelligence, pages 73–86. Springer. find
  33. Saif M Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. arXiv preprint arXiv:1708.03696. ↗
  34. Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.
  35. Omneya Rabie and Christian Sturm. 2014. Feel the heat: Emotion detection in arabic social media content. In The International Conference on Data Mining, Internet Computing, and Big Data (BigData2014), pages 37–49. The Society of Digital Information and Wireless Communication. find
  36. Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. ↗
  37. Amr M Sayed, Samir AbdelRahman, Reem Bahgat, and Aly Fahmy. 2016. Time emotional Quan Zhou, Wenlin Chen, Shiji Song, Jacob R Gardner, Kilian Q Weinberger, and Yixin Chen. 2015. A reduction of the elastic net to support vector machines with an application to gpu computing. In AAAI, pages 3210–3216. analysis of arabic tweets at multiple levels. INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 7(10):336–342. find
  38. Shadi Shaheen, Wassim El-Hajj, Hazem Hajj, and Shady Elbassuoni. 2014. Emotion recognition from text based on automatically generated rules. In Data Mining Workshop (ICDMW), 2014 IEEE International Conference on, pages 383–392. IEEE. find
  39. Amira Shoukry and Ahmed Rafea. 2012. Preprocessing egyptian dialect tweets for sentiment mining. In The Fourth Workshop on Computational Approaches to Arabic Script-based Languages, page 47.
  40. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  41. Jacopo Staiano and Marco Guerini. 2014. Depechemood: A lexicon for emotion analysis from crowd-annotated news. arXiv preprint arXiv:1405.1605. ↗
  42. Carlo Strapparava, Alessandro Valitutti, et al. 2004. Wordnet affect: an affective extension of wordnet. In LREC, volume 4, pages 1083– 1086.
  43. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307. ↗
  44. Chadi Trad, Hazem M Hajj, Wassim El-Hajj, and Fatima Al-Jamil. 2012. Facial action unit and emotion recognition with head pose variations. In ADMA, pages 383–394. Springer. find
  45. Martin Wegrzyn, Maria Vogt, Berna Kireclioglu, Julia Schneider, and Johanna Kissler. 2017. Mapping the emotional face. how individual face parts contribute to successful emotion recognition. PloS one, 12(5):e0177239. ↗
  46. Changhua Yang, Kevin Hsin-Yih Lin, and Hsin-Hsi Chen. 2007. Building emotion lexicon from weblog corpora. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 133–136. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/badaro-2018-ema-semeval-2018-task-1/