Elamur Research⌕
☰
home / papers / 2017 / A Characterization Study of Arabic Twitter Data with a Benchmarking for State-of-the-Art Opinion Mining Models

A Characterization Study of Arabic Twitter Data with a Benchmarking for State-of-the-Art Opinion Mining Models

Paper · 2017 · Ramy Baly, Gilbert Badaro, Georges El-Khoury et al.

Summary

The paper looks at why sentiment analysis of Arabic tweets is hard and how well leading English opinion-mining models work when moved to Arabic. Using the Twitter4J API with no keyword query and no language filter, the authors first gathered 610 tweets per region, with the three regions being Egypt, the Levant and the Arab Gulf. They annotated each set by hand for topic, writing style or language (MSA, dialect, Arabizi, English, other), and special tokens such as mentions, hashtags, emoticons and letter elongation. The three regions turned out to use Twitter quite differently. Next they benchmarked two families of models on the ASTD tweet dataset. The first was an SVM built on an Arabic version of the feature set from a SemEval-2016 winning system. The second was a Recursive Neural Tensor Network (RNTN) trained on ArSenTB, which is not a Twitter corpus. The RNTN with lemma input did best, and in both model families lemmas beat raw words. The results support compositional deep models and morphological abstraction for noisy, morphologically rich text.

Key points

  • Each regional set (EGY, LEV, GULF) has 610 tweets. GULF tweets are mostly in MSA, with less local dialect, and topic mixes and special-token usage differ clearly across regions.

  • The feature-engineering model reproduces, for Arabic, the features of Balikas and Amini (2016), the SemEval-2016 Task 4 five-way winner. It uses an RBF-kernel SVM, keeps n-grams only above a frequency threshold t ∈ [3, 5], and tunes the cost and γ on a dev set.

  • The RNTN uses 25-dimensional skip-gram embeddings trained on the QALB corpus (about 500K sentences). It is trained on ArSenTB and tested on ASTD tweets.

  • Results on ASTD (accuracy / average F1):

    ModelAccuracyAverage F1
    Majority baseline51.0%22.5%
    SVM baseline55.7%29.0%
    SVM all,words49.5%41.6%
    SVM all,lemmas51.7%43.4%
    RNTN words56.2%51.1%
    RNTN lemmas58.5%53.6%

  • Future work includes automatic dialect identification and cross-region evaluation, to test whether each region needs its own sentiment model.

Interesting ideas

  • I1. Before building a model, profile a sample of the target text by hand for its register traits (language mix, misspellings, tokens like hashtags and elongation), because these traits change which preprocessing matters.
  • I2. The ablation swaps words for lemmas while keeping each model fixed, which isolates the effect of morphological abstraction on sparsity.
  • I3. A sentiment model that composes meaning over a sentence tree still won on tweets after training on a non-Twitter treebank, which suggests tree-based composition transfers across domains better than surface features.
  • I4. The majority baseline gets a respectable 51% accuracy but only 22.5% average F1, so reporting both exposes models that just predict the dominant class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run a small manual profile of each new input genre (forum posts, fables, word problems) to choose test slices and to explain where the parser fails.
  • I2 (tokens): Use the same controlled word-vs-lemma swap to measure how much the lemmatizer in crate en actually helps downstream steps, instead of just assuming it does.
  • I3 (frames): Compute polarity and irony cues in prag by composing them along the UD tree, so they carry over to genres with no training data.
  • I4 (learn): Always run a majority-class negative control and report macro F1 next to accuracy for every classifier-style component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash and Khaled Shaban. 2017. A Characterization Study of Arabic Twitter Data with a Benchmarking for State-of-the-Art Opinion Mining Models. Proceedings of the Third Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1314

@inproceedings{baly2017characterization,
  title = {{A Characterization Study of Arabic Twitter Data with a Benchmarking for State-of-the-Art Opinion Mining Models}},
  author = {Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash and Khaled Shaban},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1314},
  url = {https://doi.org/10.18653/v1/w17-1314}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):12. find
  2. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  3. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan-Jan Deriu, Maurice Gonzenbach, Fatih Uzdilli, Aurelien Lucchi, Valeria De Luca, and Martin Jaggi.
  4. (SMC), 2012 IEEE International Conference on, pages 3298–3303. find
  5. IEEE. 2016. Swisscheese at semeval-2016 task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision. Proceedings of SemEval, pages 1124–1128. find
  6. Mahmoud Nabil, Mohamed A. Aly, and Amir F. Atiya. 2015. Astd: Arabic sentiment tweets dataset. In EMNLP, pages 2515–2519. ↗
  7. RM Duwairi, Raed Marji, Narmeen Sha’ban, and Sally Rushaidat. 2014. Sentiment analysis in arabic tweets. In Information and communication systems (icics), 2014 5th international conference on, pages 1–6. IEEE. find
  8. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. Semeval-2016 task 4: Sentiment analysis in twitter. Proceedings of SemEval, pages 1–18.
  9. Rasheed M. Elawady, Sherif Barakat, and Nora M. Elrashidy. 2014. Different feature selection for sentiment classification. International Journal of Information Science and Intelligent System, 3(1):137– 150. find
  10. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PloS one, 10(12):e0144296. ↗
  11. Nazlia Omar, Mohammed Albared, Adel Qasem Al-Shabi, and Tareq Al-Moslmi. 2013. Ensemble of classification algorithms for subjectivity and sentiment analysis of arabic customers’ reviews. International Journal of Advancements in Computing Technology, 5(14):77. find
  12. Ramy Eskander, Nizar Habash, Owen Rambow, and Arfath Pasha. 2016. Creating resources for dialectal arabic from a single annotation: A case study on egyptian and levantine. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3455– 3465, Osaka, Japan, December.
  13. The COLING 2016 Organizing Committee. find
  14. Arfath Pasha, Mohamed Al-Badrashiny, Mona T. Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101.
  15. Noura Farra, Elie Challita, Rawad Abou Assi, and Hazem Hajj. 2010. Sentence-level and document-level sentiment mining for arabic texts. In 2010 IEEE International Conference on Data Mining Workshops, pages 1114–1119. IEEE. find
  16. Eshrag Refaee and Verena Rieser. 2014a. Can we read emotions from a smiley face? emoticon-based distant supervision for subjectivity and sentiment analysis of arabic twitter feeds. In 5th International Workshop on Emotion, Social Signals, Sentiment and Linked Open Data, LREC. find
  17. Nizar Y. Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  18. Roula Hobeica, Hazem Hajj, and Wassim El Hajj. 2011. Machine reading for notion-based sentiment mining. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on, pages 75– 80. IEEE. find
  19. Eshrag Refaee and Verena Rieser. 2014b. Subjectivity and sentiment analysis of arabic twitter feeds with limited resources. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme, page 16. find
  20. Mohamed Maamouri, Dave Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Standard arabic morphological analyzer (sama) version 3.1. Linguistic Data Consortium, Catalog No.: LDC2010L01. find
  21. Mohammed Rushdi-Saleh, M. Teresa Martı́n-Valdivia, L. Alfonso Ureña-López, and José M. Perea-Ortega. 2011. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology, 62(10):2045–2054. ↗
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  23. Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after translation: A case-study on arabic social media posts. In HLT-NAACL, pages 767–777.
  24. Faiza Sareah. 2015. Interesting statistics for the top 10 social media sites. Small Business Trends. find
  25. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science, 34(8):1388–1429. ↗
  26. Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. J. Artif. Intell. Res.(JAIR), 55:95–130. ↗
  27. Amira Shoukry and Ahmed Rafea. 2012. Sentence-level arabic sentiment analysis. In Collaboration Technologies and Systems (CTS), 2012 International Conference on, pages 546–550. IEEE. find
  28. Asmaa Mountassir, Houda Benbrahim, and Ilham Berrada. 2012. An empirical study to address the problem of unbalanced data sets in sentiment classification. In Systems, Man, and Cybernetics Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the conference on empirical methods in natural language processing, pages 151–161. Association for Computational Linguistics.
  29. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, October. Association for Computational Linguistics.
  30. Internet World Stats. 2016. Internet world users by language. http://www.internetworldstats.com/stats7.htm. ↗
  31. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075. ↗
  32. Peter D. Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 417–424. Association for Computational Linguistics.
  33. UNESCO. 2014. World arabic language day. http://bit.ly/2lwRFYt. ↗
  34. Vdurmont. 2016. The missing emoji library for java. https://github.com/vdurmont/emoji-java. ↗
  35. Yusuke Yamamoto. 2014. Twitter4j-a java library for the twitter api. find
  36. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework. In LREC, pages 2362–2369. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baly-2017-characterization-study-arabic-twitter-data/