Elamur Research⌕
☰
home / papers / 2019 / Can Modern Standard Arabic Approaches be used for Arabic Dialects? Sentiment Analysis as a Case Study

Can Modern Standard Arabic Approaches be used for Arabic Dialects? Sentiment Analysis as a Case Study

Paper · 2019 · Kathrein Abu Kwaik, Stergios Chatzikyriakidis and Simon Dobnik

Summary

The paper asks whether sentiment classifiers built for Modern Standard Arabic (MSA) still work on dialectal Arabic (DA), or whether each dialect needs its own models. To test this, the authors create Shami-Senti, the first sentiment corpus for Levantine Arabic (Palestinian, Jordanian, Syrian, Lebanese), by annotating part of the Shami corpus. They tried labelling the corpus automatically with sentiment lexicons, found the results unreliable, and switched to native-speaker annotation. They then train classical classifiers on two other corpora, LABR (MSA book reviews) and ASTD (Egyptian tweets), and evaluate them on Shami-Senti. Across these transfer settings, accuracy stays at or below roughly 60%. Models trained on the dialect data itself perform much better. Their best feature set combines word n-grams, character n-grams and extra supporting features, and reaches 83% on MSA and 75% on Levantine. Small LSTM and BiLSTM models score lower than these feature-based classifiers. The authors conclude that MSA tools transfer poorly to dialects, so dialect-specific resources are needed.

Key points

  • Shami-Senti contains about 2,000 annotated sentences out of more than 5,000 extracted: 1,064 POS, 935 NEG and 243 mixed. Sentences are capped at 50 words, and sarcastic items were deliberately left out.
  • Lexicon-based labelling used the LABR, Moarlex and SA lexicons, and disagreed with a native annotator on up to 80% of 1,000 sentences. Two human annotators (Palestinian and Syrian) reached κ = 0.838 on 533 sentences.
  • Binary classifiers trained on LABR2 or ASTD reach only about 47–58% on Shami-Senti. The same classifiers trained in-domain on Shami-Senti reach 72–75%.
  • Classifiers tested include Ridge, LR, Passive Aggressive, LinearSVC, SGD and several Naive Bayes variants. With feature engineering, Multinomial NB reaches 75.2% on 3-class Shami-Senti.
  • The deep models used AraVec embeddings: an LSTM scored 64.7% and a BiLSTM 61.8% on Shami-Senti, below the feature-based models.

Interesting ideas

  • I1. Measuring transfer between varieties: train on one language variety, test on another, and report the gap against an in-domain model, which shows how far a "close" variety really is.
  • I2. Validating automatic labels before using them: the authors compared their lexicon-based labeller against a human on a sample, saw it fail, and dropped it instead of accepting noisy labels.
  • I3. Assigning annotators by familiarity: each annotator labelled only the dialects they knew best, which helped produce high inter-annotator agreement.
  • I4. Character n-grams for variable spelling: character n-grams added to word n-grams make a classifier more robust to dialectal spelling and morphology.
  • I5. Questioning proxy labels: the authors inspected LABR examples whose star rating marked them neutral and found that many actually carry clear polarity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Apply the same protocol to treebanks, training the parser on one English UD treebank and scoring LAS on another genre or dialect, to decide when a conversion step or separate training data is needed.
  • I2 (learn): Before accepting silver data agreed by the witness parsers, check a hand-labelled sample against it and set an agreement threshold the silver data must pass.
  • I3 (learn): Send LLM-teacher or human checks to the tier or reviewer with the most expertise in a genre, and record that assignment as part of each label's provenance.
  • I4 (tokens): Character-level features could help the tagger and lemmatizer handle informal or misspelled English without relying on a fixed vocabulary.
  • I5 (learn): When labels come from a proxy signal, audit a sample of them, and treat proxy-labelled items as weak evidence rather than gold.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Kathrein Abu Kwaik, Stergios Chatzikyriakidis and Simon Dobnik. 2019. Can Modern Standard Arabic Approaches be used for Arabic Dialects? Sentiment Analysis as a Case Study. Proceedings of the 3rd Workshop on Arabic Corpus Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kwaik2019can,
  title = {{Can Modern Standard Arabic Approaches be used for Arabic Dialects? Sentiment Analysis as a Case Study}},
  author = {Kathrein Abu Kwaik, Stergios Chatzikyriakidis and Simon Dobnik},
  booktitle = {Proceedings of the 3rd Workshop on Arabic Corpus Linguistics},
  year = {2019},
  url = {https://aclanthology.org/W19-5606/}
}

References

47 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2012. Toward building a large-scale Arabic sentiment lexicon. In Proceedings of the 6th international global WordNet conference, pages 18–22. find
  2. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. SAMAR: Subjectivity and sentiment analysis for Arabic social media. Computer Speech & Language, 28(1):20–37. ↗
  3. Muhammad Abdul-Mageed and Mona T Diab. 2011. Subjectivity and sentiment annotation of modern standard Arabic newswire. In Proceedings of the 5th linguistic annotation workshop, pages 110–118. Association for Computational Linguistics.
  4. Muhammad Abdul-Mageed, Mona T Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 587–591. Association for Computational Linguistics.
  5. Nawaf A Abdulla, Nizar A Ahmed, Mohammed A Shehab, and Mahmoud Al-Ayyoub. 2013. Arabic sentiment analysis: Lexicon-based and corpus-based. In 2013 IEEE Jordan conference on applied electrical engineering and computing technologies (AEECT), pages 1–6. IEEE. find
  6. Malak Abdullah and Mirsad Hadzikadic. 2017. Sentiment analysis on Arabic tweets: Challenges to dissecting the language. In International Conference on Social Computing and Social Media, pages 191– 202. Springer. find
  7. Hamed Al-Rubaiee, Renxi Qiu, and Dayou Li. 2016. Identifying Mubasher software products through sentiment analysis of Arabic tweets. In 2016 International Conference on Industrial Informatics and Computer Systems (CIICS), pages 1–6. IEEE. find
  8. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for Arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on Arabic Natural Language Processing (ANLP), pages 165–173.
  9. Gilbert Badaro, Hussein Jundi, Hazem Hajj, Wassim El-Hajj, and Nizar Habash. 2018. ArSEL: A Large Scale Arabic Sentiment and Emotion Lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  10. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Shaban. 2017a. A characterization study of Arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In Proceedings of the third Arabic natural language processing workshop, pages 110–118.
  11. Ramy Baly, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Khaled Bashir Shaban, and Wassim El-Hajj. 2017b. Comparative evaluation of sentiment analysis methods across Arabic dialects. Procedia Computer Science, 117:266–273. find
  12. Jean Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 2(22):249–254. ↗
  13. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7(Mar):551–585. find
  14. Harris Drucker, Christopher JC Burges, Linda Kaufman, Alex J Smola, and Vladimir Vapnik. 1997. Support vector regression machines. In Advances in neural information processing systems, pages 155– 161. find
  15. Nora Al-Twairesh, Hend S. Al-Khalifa, AbdulMalik Al-Salman, and Yousef Al-Ohali. 2018. Sentiment Analysis of Arabic Tweets: Feature Engineering and A Hybrid Approach. CoRR, abs/1805.08533. ↗
  16. Rehab M Duwairi, Raed Marji, Narmeen Sha’ban, and Sally Rushaidat. 2014. Sentiment analysis in Arabic tweets. In 2014 5th International Conference on Information and Communication Systems (ICICS), pages 1–6. IEEE. find
  17. Mohamed Aly and Amir Atiya. 2013. Labr: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 494–498. ↗
  18. Alaa M El-Halees. 2011. Arabic opinion mining using combined classification approach. Arabic opinion mining using combined classification approach. find
  19. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A Survey of Opinion Mining in Arabic: A Comprehensive System Perspective Covering Challenges and Advances in Tools, Resources, Models, Applications, and Visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TAL-LIP), 18(3):27. find
  20. Hady ElSahar and Samhaa R El-Beltagy. 2014. A fully automated approach for Arabic slang lexicon extraction from microblogs. In International conference on intelligent text processing and computational linguistics, pages 79–91. Springer. find
  21. Umar Farooq, Hasan Mansoor, Antoine Nongaillard, Yacine Ouzrout, and Muhammad Abdul Qadir. 2017. Negation Handling in Sentiment Analysis at Sentence Level. JCP, 12(5):470–478. ↗
  22. Noura Farra, Elie Challita, Rawad Abou Assi, and Hazem Hajj. 2010. Sentence-level and document-level sentiment mining for Arabic texts. In 2010 IEEE international conference on data mining workshops, pages 1114–1119. IEEE. find
  23. Jiashi Feng, Huan Xu, Shie Mannor, and Shuicheng Yan. 2014. Robust logistic regression and classification. In Advances in neural information processing systems, pages 253–261. find
  24. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 55–64. word, Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519.
  25. Donia Gamal, Marco Alfonse, El-Sayed M. El-Horbaty, and Abdel-Badeeh M.Salem. 2018. Opinion Mining for Arabic Dialects on Twitter. Egyptian Computer Science Journal, 42(4):52–61. find
  26. Ahmed Oussous, Ayoub Ait Lahcen, and Samir Belfkih. 2018. Improving Sentiment Analysis of Moroccan Tweets Using Ensemble Learning. In International Conference on Big Data, Cloud and Applications, pages 91–104. Springer. find
  27. Charles A Ferguson. 1959. 15(2):325–340. Diglossia. find
  28. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 161–169. ↗
  29. Tobias Günther and Lenz Furrer. 2013. GU-MLT-LT: Sentiment analysis of short messages using linguistic features and stochastic gradient descent. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), volume 2, pages 328–332. find
  30. Hossam S Ibrahim, Sherif M Abdou, and Mervat Gheith. 2015. Sentiment analysis for Modern Standard Arabic and colloquial. arXiv preprint arXiv:1505.03105. ↗
  31. Jihen Karoui, Farah Banamara Zitoune, and Veronique Moriceau. 2017. Soukhria: Towards an irony detection system for Arabic in social media. Procedia Computer Science, 117:161–168. ↗
  32. Suresh Kumar and Shivani Goel. 2015. Enhancing Text Classification by Stochastic Optimization method and Support Vector Machine. International Journal of Computer Science and Information Technologies, 6 (4), pages 3742–3745. find
  33. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnika. 2018. A Lexical Distance Study of Arabic Dialects. Procedia computer science, 142:2–13. find
  34. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  35. Chatrine Qwaider, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. Shami: A corpus of levantine arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
  36. Kumar Ravi and Vadlamani Ravi. 2015. A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowledge-Based Systems, 89:14–46. ↗
  37. Rizkallah, Sandra and Atiya, Amir and ElDin Mahgoub, Hossam and Heragy, Momen”, editor=”Hassanien, Aboul Ella and Tolba, Mohamed F. and Elhoseny, Mohamed and Mostafa, Mohamed . 2018. Dialect Versus MSA Sentiment Analysis. In The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2018), pages 605–613, Cham. Springer International Publishing. find
  38. Lina Maria Rojas-Barahona. 2016. Deep learning for sentiment analysis. Language and Linguistics Compass, 10(12):701–719. ↗
  39. Hiroshi Shimodaira. 2014. Text classification using naive bayes. Learning and Data Note, 7:1–9. find
  40. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. ↗
  41. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of Arabic word embedding models for use in Arabic nlp. Procedia Computer Science, 117:256–265. find
  42. Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments. In Proceedings of the third Arabic natural language processing workshop, pages 55–61.
  43. Duyu Tang, Bing Qin, and Ting Liu. 2015. Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(6):292–303. ↗
  44. Kees Versteegh. 2014. The Arabic language. Edinburgh University Press. ↗
  45. Shuo Xu, Yan Li, and Zheng Wang. 2017. Bayesian multinomial Naı̈ve Bayes classifier to text classification. In Advanced multimedia and ubiquitous engineering, pages 347–352. Springer. find
  46. Mohab Youssef and Samhaa R El-Beltagy. 2018. MoArLex: An Arabic Sentiment Lexicon Built Through Automatic Lexicon Expansion. Procedia computer science, 142:94–103. find
  47. Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kwaik-2019-can-modern-standard-arabic-approaches/