Elamur Research⌕
☰
home / papers / 2020 / Toward Qualitative Evaluation of Embeddings for Arabic Sentiment Analysis

Toward Qualitative Evaluation of Embeddings for Arabic Sentiment Analysis

Summary

Arabic is agglutinative and morphologically rich: one surface word can carry several clitics, and missing diacritics hide meanings with different polarity. This spreads vocabulary thin and can weaken embeddings. The authors ask how to judge embeddings specifically for Arabic sentiment analysis, and whether lemma-based vectors beat word-based ones. They trained word2vec models at both word and lemma level on three kinds of corpora: polar (opinion text), non-polar (generic text) and a mix of the two. They also tracked how the models change across training epochs. For intrinsic evaluation they introduce a sentiment-specific test: does a polar unit's neighbourhood in the vector space keep the same polarity? For extrinsic evaluation they feed the embeddings to a Kim-style CNN, with the embeddings either frozen (static) or fine-tuned (non-static). Experiments use binary positive/negative classification on the LABR book-review corpus. The best system reaches 91.9%, above the previously published 91.5%. The embeddings and evaluation resources are released for free use, which helps Arabic NLP research where resources are scarce.

Key points

  • Lexical units compared are word and lemma. Lemmatization folds gender and number variants into one entry, which shrinks the vocabulary and reduces sparsity. Earlier work had already found lemmas the strongest of six Arabic units.
  • The embedding corpora vary by type (polar, non-polar, mixed) and by size. The number of training epochs is studied for its effect on the neighbours of polar units.
  • The CNN uses filter windows of 3, 4 and 5, then max pooling, a fully connected layer with dropout, and a sigmoid output. It runs in static (frozen embeddings) and non-static (trainable embeddings) modes.
  • The paper tests a specific trade-off: frozen embeddings trained on in-domain polar text versus generic embeddings that are adapted during task training.
  • On LABR binary classification the best system scores 91.9%, against a previous best of 91.5%. The pre-trained embeddings and evaluation resources are publicly released.

Interesting ideas

  • I1. Neighbour-stability probe: Judge a vector space by checking whether a word's nearest neighbours share a task-relevant property, here polarity. This measures what the task needs rather than generic analogy scores.
  • I2. Tracking neighbourhoods across epochs: Inspect how the neighbours of key units drift as training continues, instead of only scoring the final model. This exposes instability and helps choose a stopping point.
  • I3. Lemma as the representation unit: Building vectors over lemmas instead of surface forms pools evidence from inflected variants. This fights sparsity in morphologically rich text.
  • I4. Separating "in-domain pretraining" from "task fine-tuning": The study frames domain-matched frozen vectors and generic vectors adapted during training as two distinct options. Comparing them shows where the gain actually comes from.
  • I5. Controlled corpus-type variation: Training identical models on polar, non-polar and mixed corpora isolates how the data source affects the result.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Check learned lexical vectors or verb-class embeddings by testing whether neighbours keep the same VerbNet class or frame. Neighbours that cross classes would flag a regression.
  • I2 (learn): While training gparse or the tagger, snapshot the representations of a fixed probe set of words each epoch. Report how much their neighbourhoods drift, as an explainable training diagnostic.
  • I3 (tokens): Key any lexical statistics or embeddings in crate en on lemma plus UPOS rather than wordform. Rare inflections would then inherit evidence from their lemma.
  • I4 (learn): Run the same controlled comparison when deciding between silver data from fable/tale text and generic treebanks fine-tuned on the target. This keeps the source of improvement attributable.
  • I5 (tree): Train the perceptron parser on different treebank mixes under fixed settings. Measure LAS per mix to see which genres help narrative text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Sentiment & emotion Evaluation & benchmarks Semitic languages

 Institutions

🔬Laboratoire d’Informatique de l’Université du MansLe Mans · France7 works · 2,737 citations🔬Laboratoire Informatique d'AvignonAvignon · France4 works · 414 citations

Cite

Cite as Amira Barhoumi, Nathalie Camelin, Chafik Aloulou, Yannick Estève and Lamia Hadrich Belguith. 2020. Toward Qualitative Evaluation of Embeddings for Arabic Sentiment Analysis. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{barhoumi2020toward,
  title = {{Toward Qualitative Evaluation of Embeddings for Arabic Sentiment Analysis}},
  author = {Amira Barhoumi, Nathalie Camelin, Chafik Aloulou, Yannick Estève and Lamia Hadrich Belguith},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.610/}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Ayyoub, M., Khamaiseh, A. A., Jararweh, Y., and Al-Kabi, M. N. (2018). A comprehensive survey of arabic sentiment analysis. Information Processing & Management. find
  2. Al-Ayyoub, M., Khamaiseh, A. A., Jararweh, Y., and Al-Kabi, M. N. (2019). A comprehensive survey of arabic sentiment analysis. Information Processing & Management, 56(2):320–342. find
  3. Al-Moslmi, T., Albared, M., Al-Shabi, A., Omar, N., and Abdullah, S. (2018). Arabic senti-lexicon: Constructing publicly available language resources for arabic sentiment analysis. Journal of Information Science, 44(3):345–362. find
  4. Al Sallab, A., Hajj, H., Badaro, G., Baly, R., El Hajj, W., and Shaban, K. B. (2015). Deep learning models for sentiment analysis in arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 9–17. find
  5. Al-Smadi, M., Talafha, B., Al-Ayyoub, M., and Jararweh, Y. (2018). Using long short-term memory deep neural networks for aspect-based sentiment analysis of arabic reviews. International Journal of Machine Learning and Cybernetics, pages 1–13. find
  6. Antoniak, M. and Mimno, D. (2018). Evaluating the stability of embedding-based word similarities. Transactions of the Association for Computational Linguistics, 6:107– 119. find
  7. Badaro, G., Baly, R., Hajj, H., Habash, N., and El-Hajj, W. (2014). A large scale arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on arabic natural language processing (ANLP), pages 165–173. find
  8. Badaro, G., Baly, R., Hajj, H., El-Hajj, W., Shaban, K. B., Habash, N., Al-Sallab, A., and Hamdi, A. (2019). A survey of opinion mining in arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(3):27. find
  9. Barhoumi, A., Camelin, N., and Estève, Y. (2018). Des représentations continues de mots pour l’analyse d’opinions en arabe: une étude qualitative. In 25e conférence sur le Traitement Automatique des Langues Naturelles (TALN 2018), Rennes, France, May. find
  10. Barhoumi, A., Camelin, N., Aloulou, C., Estève, Y., and Hadrich Belguith, L. (2019). An empirical evaluation of arabic-specific embeddings for sentiment analysis. In Kamel Smaı̈li, editor, Arabic Language Processing: From Theory to Practice, pages 34–48, Cham. Springer International Publishing. find
  11. Baroni, M., Dinu, G., and Kruszewski, G. (2014). Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238–247. our card
  12. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2016). Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗ our card
  13. Dahou, A., Xiong, S., Zhou, J., Haddoud, M. H., and Duan, P. (2016). Word embeddings and convolutional neural network for arabic sentiment classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2418–2427. find
  14. El-Khair, I. A. (2016). 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  15. El Mahdaouy, A., El Alaoui, S. O., and Gaussier, E. (2018). Improving arabic information retrieval using word embedding similarities. Int. J. Speech Technol., 21(1):121– 136, March. find
  16. Elnagar, A. and Einea, O. (2016). Brad 1.0: Book reviews in arabic dataset. In Computer Systems and Applications (AICCSA), 2016 IEEE/ACS 13th International Conference of, pages 1–8. IEEE. find
  17. Elnagar, A., Khalifa, Y. S., and Einea, A. (2018). Hotel arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer. find
  18. Elrazzaz, M., Elbassuoni, S., Shaban, K., and Helwe, C. (2017). Methodical evaluation of Arabic word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 454–458, Vancouver, Canada, July. Association for Computational Linguistics. find
  19. ElSahar, H. and El-Beltagy, S. R. (2015). Building large arabic multi-domain resources for sentiment analysis. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 23–34. Springer. find
  20. Fouad, M. M., Mahany, A., Aljohani, N., Abbasi, R. A., and Hassan, S.-U. (2019). Arwordvec: efficient word embedding models for arabic tweets. Soft Computing, Jun. find
  21. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). our card
  22. Hassan, A. (2017). Sentiment analysis with recurrent neural network and unsupervised neural language model. find
  23. Heikal, M., Torki, M., and El-Makky, N. (2018). Sentiment analysis of arabic tweets using deep learning. Procedia Computer Science, 142:114–122. find
  24. Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. find
  25. Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗ our card
  26. Kiritchenko, S., Zhu, X., and Mohammad, S. M. (2014). Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. find
  27. Lachraf, R., Billah Nagoudi, E. M., Ayachi, Y., Abdelali, A., and Schwab, D. (2019). ArbEngVec : Arabic-English Cross-Lingual Word Embedding Model. In The Fourth Arabic Natural Language Processing Workshop, Florence, Italy, July. find
  28. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111– 3119. our card
  29. Mohammad, S. M. and Turney, P. D. (2010). Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, pages 26–34. Association for Computational Linguistics. our card
  30. Mohammad, S. and Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. CoRR, abs/1308.6297. ↗
  31. Mohammad, S. M. and Yang, T. W. (2011). Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis, pages 70–79. Association for Computational Linguistics. find
  32. Mohammad, S. M., Kiritchenko, S., and Zhu, X. (2013). Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. ↗ our card
  33. Nabil, M., Aly, M., and Atiya, A. (2014). Labr: A large scale arabic sentiment analysis benchmark. arXiv preprint arXiv:1411.6718. ↗
  34. Nasukawa, T. and Yi, J. (2003). Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture, pages 70–77. ACM. find
  35. Nayak, N., Angeli, G., and Manning, C. D. (2016). Evaluating word embeddings using a representative suite of practical tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 19–23. find
  36. Pang, B., Lee, L., et al. (2008). Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. find
  37. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proc. of NAACL. our card
  38. Pierrejean, B. and Tanguy, L. (2018). Towards qualitative word embeddings evaluation: Measuring neighbors variation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 32–39, New Orleans, Louisiana, USA, June. Association for Computational Linguistics. find
  39. Saif M. Mohammad, M. S. and Kiritchenko, S. (2016). Sentiment lexicons for arabic social media. In Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  40. Salama, R. A., Youssef, A., and Fahmy, A. (2018). Morphological word embedding for arabic. Procedia computer science, 142:83–93. find
  41. Schnabel, T., Labutov, I., Mimno, D., and Joachims, T. (2015). Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 298–307. our card
  42. Shapiro, P. and Duh, K. (2018). Morphological word embeddings for Arabic neural machine translation in low-resource settings. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 1–11, New Orleans, June. Association for Computational Linguistics. our card
  43. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  44. Wilson, T., Wiebe, J., and Hoffmann, P. (2005). Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barhoumi-2020-qualitative-evaluation-embeddings-arabic-sentiment/