Elamur Research⌕
☰
home / papers / 2020 / An Arabic Tweets Sentiment Analysis Dataset (ATSAD) using Distant Supervision and Self Training

An Arabic Tweets Sentiment Analysis Dataset (ATSAD) using Distant Supervision and Self Training

Paper · 2020 · Kathrein Abu Kwaik, Stergios Chatzikyriakidis, Simon Dobnik et al.

Summary

Arabic sentiment analysis lacks social-media resources that are both large and reliably labelled. Existing corpora are often small, limited to one dialect, unbalanced, or labelled in ways that distort the original text. The authors build ATSAD, a Twitter corpus of 36K Arabic tweets marked as positive or negative. They gathered tweets with the Twitter API, using a sentiment-tagged emoji lexicon as the search seed. Each tweet then received the polarity of the emoji that retrieved it, a form of distant supervision with emojis as weak, noisy labels. Two annotators hand-labelled an 8K subset, which serves as a gold standard and test set. To clean up the automatic labels, the authors add self-training and treat it as a second check on the emoji-derived labels. They evaluate the corpus intrinsically against human labels and pre-trained models, and extrinsically by training classifiers. The best setup reaches 86% accuracy. The work shows that a cheap weak-labelling pipeline, anchored by a modest gold set, can produce a usable multi-dialect resource.

Key points

  • The seed lexicon merges the emoji sentiment ranking published by Kralj Novak et al. (2015) with Emojipedia, yielding 306 positive and 91 negative emojis.
  • About 59K Arabic tweets were collected in April 2019 with no geographic filter, so many dialects are covered. After pre-processing, 36K labelled tweets remain.
  • The 8K manually annotated tweets agree with the emoji-derived labels 77.2% of the time.
  • Two models were trained: a unigram baseline and a richer model using word n-grams plus character n-grams. Self-training on top of distant supervision reaches 86% accuracy.
  • The design follows the English Stanford Twitter Sentiment (STS) dataset, which has 1.6M emoticon-labelled tweets and a small manual test set. The data is released on GitHub (motazsaad/arabic-sentiment-analysis).

Interesting ideas

  • I1. The emoji lexicon serves both as the search query and as the label source, so collecting the data and labelling it happen in a single pass.
  • I2. Before relying on weak labels, the authors measure their noise against a small human-labelled subset and report it openly (77.2%).
  • I3. A classifier trained through self-training acts as a second opinion on the heuristic labels, a "double check" instead of trusting one weak signal.
  • I4. The authors criticise a competing corpus for cleaning so heavily that it no longer looks like real tweets; normalisation should keep the writer's style.
  • I5. Emojis are treated as explicit sentiment markers attached to a text, separate from its words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A seed list of unambiguous cues (e.g. verbs strongly tied to a VerbNet class) could both retrieve sentences and pre-label them, cheaply bootstrapping silver training data.
  • I2 (learn): Every silver-data source should get a measured agreement rate against a small gold sample, so its reliability is a known number rather than an assumption.
  • I3 (learn): This fits the witness-parser scheme, where a trained model's vote is pooled with rule-derived labels and only agreeing items are promoted to silver.
  • I4 (tokens): The tokenizer should store normalised forms next to the original surface text, so later stages can still see spelling, repetition and style.
  • I5 (frames): Pragmatics could record emojis and emoticons as speaker-attitude signals on the speech act, which can also help flag irony when they clash with the literal content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Kathrein Abu Kwaik, Stergios Chatzikyriakidis, Simon Dobnik, Motaz Saad and Richard Johansson. 2020. An Arabic Tweets Sentiment Analysis Dataset (ATSAD) using Distant Supervision and Self Training. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kwaik2020arabic,
  title = {{An Arabic Tweets Sentiment Analysis Dataset (ATSAD) using Distant Supervision and Self Training}},
  author = {Kathrein Abu Kwaik, Stergios Chatzikyriakidis, Simon Dobnik, Motaz Saad and Richard Johansson},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.1/}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Diab, M., and Kübler, S. (2014). Samar: Subjectivity and sentiment analysis for Arabic social media. Computer Speech & Language, 28(1):20– 37. ↗
  2. Abney, S. (2002). Bootstrapping. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 360–367. find
  3. Al Sallab, A., Hajj, H., Badaro, G., Baly, R., El Hajj, W., and Shaban, K. B. (2015). Deep learning models for sentiment analysis in Arabic. In Proceedings of the second workshop on Arabic natural language processing, pages 9–17. ↗
  4. Al Shboul, B., Al-Ayyoub, M., and Jararweh, Y. (2015). Multi-way sentiment classification of Arabic reviews. In 2015 6th International Conference on Information and Communication Systems (ICICS), pages 206–211. IEEE. find
  5. Al-Twairesh, N., Al-Khalifa, H., and Al-Salman, A. (2016). AraSenTi: Large-Scale Twitter-Specific Arabic Sentiment Lexicons. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 697–705, Berlin, Germany, August. Association for Computational Linguistics.
  6. Al-Twairesh, N., Al-Khalifa, H., Al-Salman, A., and Al-Ohali, Y. (2017). Arasenti-tweet: A corpus for Arabic sentiment analysis of saudi tweets. Procedia Computer Science, 117:63–72. find
  7. Alayba, A. M., Palade, V., England, M., and Iqbal, R. (2018). A combined CNN and LSTM model for Arabic sentiment analysis. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 179–191. Springer.
  8. Alwakid, G., Osman, T., and Hughes-Roberts, T. (2017). Challenges in sentiment analysis for Arabic social networks. Procedia Computer Science, 117:89–100. find
  9. Aly, M. and Atiya, A. (2013). Labr: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 494–498. ↗
  10. Atoum, J. O. and Nouman, M. (2019). Sentiment Analysis of Arabic Jordanian Dialect Tweets. (IJACSA) International Journal of Advanced Computer Science and Applications, 10:256–262. find
  11. Azmi, A. M. and Alzanin, S. M. (2014). Aara’–a system for mining the polarity of Saudi public opinion through e-newspaper comments. Journal of Information Science, 40(3):398–410. ↗
  12. Bakliwal, A., Arora, P., Madhappan, S., Kapre, N., Singh, M., and Varma, V. (2012). Mining sentiments from tweets. In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pages 11–18. find
  13. Bravo-Marquez, F., Mendoza, M., and Poblete, B. (2013). Combining strengths, emotions and polarities for boosting twitter sentiment analysis. In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, page 2. ACM. find
  14. Dahou, A., Xiong, S., Zhou, J., Haddoud, M. H., and Duan, P. (2016). Word embeddings and convolutional neural network for Arabic sentiment classification. In Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers, pages 2418–2427.
  15. El-Beltagy, S. R., Khalil, T., Halaby, A., and Hammad, M. (2016). Combining lexical features and a supervised learning approach for Arabic sentiment analysis. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 307–319. Springer. find
  16. Elnagar, A., Khalifa, Y. S., and Einea, A. (2018). Hotel Arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer. ↗
  17. Gao, W., Li, S., Xue, Y., Wang, M., and Zhou, G. (2014). Semi-supervised sentiment classification with self-training on feature subspaces. In Workshop on Chinese Lexical Semantics, pages 231–239. Springer. find
  18. Giachanou, A. and Crestani, F. (2016). Like It or Not: A Survey of Twitter Sentiment Analysis Methods. ACM Comput. Surv., 49(2):28:1–28:41, June. find
  19. Go, A., Bhayani, R., and Huang, L. (2009). Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, 1(12):2009. find
  20. Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., and Weld, D. S. (2011). Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 541–550. Association for Computational Linguistics. find
  21. Kralj Novak, P., Smailović, J., Sluban, B., and Mozetič, I. (2015). Sentiment of emojis. PLoS ONE, 10(12):e0144296. ↗
  22. Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. ↗
  23. Mohammed, A. and Kora, R. (2019). Deep learning approaches for Arabic sentiment analysis. Social Network Analysis and Mining, 9(1):52. ↗
  24. Nabil, M., Aly, M., and Atiya, A. (2015). ASTD: Arabic Sentiment Tweets Dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519. ↗
  25. Obaidat, I., Mohawesh, R., Al-Ayyoub, M., Mohammad, A.-S., and Jararweh, Y. (2015). Enhancing the determination of aspect categories and their polarities in Arabic reviews using lexicon-based approaches. In 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT), pages 1–6. IEEE. find
  26. Qwaider, C., Chatzikyriakidis, S., and Dobnik, S. (2019). Can Modern Standard Arabic Approaches be used for Arabic Dialects? Sentiment Analysis as a Case Study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 40–50.
  27. Refaee, E. and Rieser, V. (2014). Evaluating distant supervision for subjectivity and sentiment analysis on Arabic twitter feeds. In Proceedings of the EMNLP 2014 workshop on Arabic natural language processing (ANLP), pages 174–179.
  28. Resnik, P. and Lin, J. (2010). 11. evaluation of NLP systems. The handbook of computational linguistics and natural language processing, 57. find
  29. Rushdi-Saleh, M., Martı́n-Valdivia, M. T., Ureña-López, L. A., and Perea-Ortega, J. M. (2011). OCA: Opinion Corpus for Arabic. Journal of the American Society for Information Science and Technology, 62(10):2045– 2054. ↗
  30. Saif, H., He, Y., and Alani, H. (2012). Alleviating data sparsity for twitter sentiment analysis. CEUR Workshop Proceedings (CEUR-WS. org). find
  31. Speriosu, M., Sudan, N., Upadhyay, S., and Baldridge, J. (2011). Twitter polarity classification with label propagation over lexical links and the follower graph. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 53–63. Association for Computational Linguistics. find
  32. Yao, L., Riedel, S., and McCallum, A. (2010). Collective cross-document relation extraction without labelled data. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1013–1023. Association for Computational Linguistics. find
  33. Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189–196.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kwaik-2020-arabic-tweets-sentiment-analysis-dataset/