Elamur Research⌕
☰
home / papers / 2021 / An Enhanced Corpus for Arabic Newspapers Comments

An Enhanced Corpus for Arabic Newspapers Comments

Summary

The authors build a sentiment resource for reader comments on the websites of three well-known Algerian Arabic-language newspapers. These comments are mostly in MSA but often contain Algerian dialect words and spellings, such as writing د in place of ذ, a trait of the Algiers dialect. They extend an earlier approach in two ways: the corpus is larger, and annotation is folded into the MATTER cycle (Model, Annotate, Train, Test, Evaluate, Revise). Their variant, called MApTTER, adds explicit adjudication and processing steps. Two annotators label the comments. A gold standard is then adjudicated, the neutral class is dropped, and the data is balanced. SVM, naïve Bayes and KNN are compared with 10-fold cross-validation over several feature setups: unigram, bigram and trigram features; TO, TF, TF-IDF and BTO weightings; and runs with and without light stemming. Accuracies land roughly in the 56–75% range. Light stemming brings no clear gain, which the authors attribute to dialectal vocabulary, and trigrams do not help. The work matters as a rare news-comment resource for a dialect-heavy, under-resourced setting.

Key points

  • Source data: comments from the websites of three Algerian newspapers, labelled positive or negative through a MATTER-style annotate–evaluate–revise loop.
  • In the second round, annotators were told to judge only the comment's own content. Inter-annotator agreement rose to kappa = 0.5820.
  • The second-round gold standard was balanced at 194 negative and 194 positive comments, with neutral comments removed.
  • In the second round, naïve Bayes peaked at 75.00% accuracy (bigram, TF, light stemming) and reached 93.30% negative-class recall without stemming. SVM did worse than in round one despite the better agreement.
  • Light stemming gave no considerable improvement, and the trigram model was judged a poor representation. Planned next steps are focusing on a comment's first and last parts, handling dialectal and French segments, and trying deep learning.

Interesting ideas

  • I1. Annotation is run as an explicit loop: measure agreement, find the source of disagreement, change the annotator instructions, and re-annotate. Here, limiting judgments to the comment itself measurably raised kappa, which shows guideline fixes can be tested like model changes. — For Mova (learn): When LLM teacher tiers or witness parsers disagree on silver labels, Mova could revise the prompt or guideline and re-measure agreement before training, logging each revision as one cycle.
  • I2. The authors pair improved agreement with worse SVM results, a reminder that cleaner labels do not automatically mean better classifiers. Both quantities should be reported side by side. — For Mova (learn): Evaluation reports could show teacher agreement next to downstream accuracy, so a "cleaner" silver set that hurts the parser is caught instead of assumed beneficial.
  • I3. Stemming failed here because dialectal word forms escape a standard-language stemmer. A normaliser built for the standard variety can therefore be useless on non-standard text, and its effect should be ablated rather than assumed. — For Mova (tokens): The en crate's lemmatiser could be ablated on informal or dialectal English, such as forum comments, to check whether lemmatisation actually helps tagging there.
  • I4. The paper flags the first and last parts of a comment as carrying most of its meaning. This position-aware view can weight opening and closing statements more heavily. — For Mova (frames): When deciding a text's stance or speech act, prag could give extra weight to frames from opening and closing sentences, where evaluative statements tend to sit.
  • I5. The proposed fix for mixed-language comments, with dialect and French segments inside Arabic, is to detect and treat those segments separately instead of discarding them. — For Mova (tokens): The tokenizer could tag embedded foreign-language spans so the parser handles them as opaque units rather than mis-tagging them as English words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.34028/iajit/17/5/12 · arXiv 2102.09965

@misc{rahab2021enhanced,
  title = {{An Enhanced Corpus for Arabic Newspapers Comments}},
  author = {Rahab, Hichem and Zitouni, Abdelhafid and Djoudi, Mahieddine},
  year = {2021},
  doi = {10.34028/iajit/17/5/12},
  eprint = {2102.09965},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.34028/iajit/17/5/12}
}

References

37 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed M. and Diab M., “AWATIF: A Multi-Genre Corpus for Modern Standard Arabic negative class without stemming, as shown in Table 5. Bigram Precision Pos Neg 82,14 58,88 89,66 56,97 90,91 53,80 89,06 57,72 79,61 60,70 69,94 66,05 66,33 66,67 81,01 57,93 Tri-gram Precision Recall Pos Neg Pos Neg 81,93 58,69 35,05 92,27 89,83 57,14 27,32 96,91 91,67 54,26 17,01 98,45 88,71 57,36 28,35 96,39 79,61 60,70 42,27 89,18 69,77 65,74 61,86 73,20 66,33 66,67 67,01 65,98 81,01 57,93 32,99 92,27 Recall Pos Neg 35,57 92,27 26,80 96,91 15,46 98,45 29,38 96,39 42,27 89,18 62,37 73,20 67,01 65,98 32,99 92,27 Acc 63,66 62,11 57,73 62,37 65,72 67,53 66,49 62,63 Subjectivity and Sentiment Analysis,” in Proceedings of Language Resources and Evaluation Conference, Istanbul, pp. 3907-3914, 2012. find
  2. Al-Harbi O., “Classifying Sentiment Of Dialectal Arabic Reviews: A Semi-Supervised Approach,” The International Arab Journal of Information Technology, vol. 16, no. 6, pp. 995–1002, 2019. find
  3. Al-Kabi M. N., Al-ayyoub M., Alsmadi I., and Wahsheh H., “A Prototype for a Standard Arabic Sentiment Analysis Corpus,” International Arab Journal of Information Technology, vol. 13, no. 2, pp. 163-170, 2016. find
  4. Al-Kabi M., Al-qwaqenah A., Gigieh A., Alsmearat K., Al-ayyoub M., and Alsmadi I., “Building a Standard Dataset for Arabic Sentiment Analysis,” in Proceedings of the IEEE/ACS 13 th International Conference of Computer Systems and Applications,Agadir, pp. 1-6, 2016. find
  5. Alotaibi S. and Anderson C., “Extending the Knowledge of the Arabic Sentiment Classification Using a Foreign External Lexical Source,” International Journal on Natural Language Computing, vol. 5, no. 3, pp. 1-11, 2016. find
  6. Atia S. and Shaalan K., “Increasing the accuracy of opinion mining in Arabic,” in Proceedings of Conference: International Conference on Arabic Computational Linguistics, Cairo, pp. 106-113, 2015. find
  7. Badaro G., Baly R., Hajj H., El-Hajj W., Shaban K ., Habash N ., Al-Sallab A., and Hamdi A., “A Survey of Opinion Mining in Arabic: A Comprehensive System Perspective Covering Challenges and Advances in Tools, Resources, Models, Applications, and Visualizations,” ACM Transactions on Asian Language Information Processing, vol. 18, no. 3, pp. 1-52, 2019. find
  8. Ben-Hur A. and Weston J., “A user’s guide to support vector machines.,” Methods in molecular biology, vol. 609, pp. 223–239, 2010. find
  9. Bougrine S., Cherroun H., and Abdelali A., “Altruistic Crowdsourcing for Arabic Speech Corpus Annotation,” Procedia Computer Science, vol. 117, pp. 133-144, 2017. find
  10. Brahimi B., Touahria M., and Tari A., “Data and Text Mining Techniques for Classifying Arabic Tweet Polarity,” Journal of Digital Information Management, vol. 14, no. 1, pp. 15-25, 2016. find
  11. Carletta J., “Assessing agreement on classification tasks: the kappa statistic,” Computational Linguistics, vol. 22, no. 2, pp. 249-254, 1996. find our card
  12. Cherif W., Madani A., and Kissi M., “Towards an Efficient Opinion Measurement in Arabic Comments,” Procedia Comput Science, vol. 73, pp. 122-129, 2015. find
  13. Cortes C. and Vapnik V., “Support-Vector Networks,” Machine Learning, vol. 20, no. 3, pp. 273-297, 1995. find
  14. Elnagar A. and Einea O., “BRAD 1 . 0 : Book Reviews in Arabic Dataset,” in Proceedings of IEEE/ACS 13 th International Conference of Computer Systems and Applications, Agadir, pp. 1-8, 2016. find
  15. Elnagar A., Khalifa Y., and Einea A., Intelligent Natural Language Processing: Trends and Applications, in Springer International Publishing , 2018. find
  16. Guellil I., Adeel A., Azouaou F., and Hussain A., “SentiALG : Automated Corpus Annotation for Algerian Sentiment Analysis Introduction,” in Proceedings of the International Conference on Brain Inspired Cognitive Systems, Xi'an, pp. 557-567, 2018 find
  17. Habash N., Soudi A., and Buckwalter T., “On Arabic Transliteration,” In Arabic Computational Morphology: Knowledge-based and Empirical Methods, vol. 49, no. 4, pp. 15-22, 2007. find
  18. Hamdi A., Shaban K., and Zainal A., “CLASENTI : A class-specific sentiment analysis framework,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 17, no. 4, p. 32, 2018. find
  19. Harrat S., Meftouh K., Abbas M., Hidouci K., and Smaili K., “An Algerian Dialect : Study and Resources,” International Journal of Advanced Computer Science and Applications, vol. 7, no. 3, pp. 384-396, 2016. find
  20. Ibrahim H. S., Abdou S., and Gheith M., “MIKA: A Tagged Corpus for Modern Standard Arabic and Colloquial Sentiment Analysis,” in Proceedings of IEEE 2 nd International Conference on Recent Trends in Information Systems, Kolkata, pp 353-358, 2015. find
  21. Ide N. and Pustejovsky J., Handbook of Linguistic Annotation, Springer, 2017. find
  22. Jarrar M., Habash N., Alrimawi F., Akra D., and Zalmout N., “Curras: an annotated corpus for the Palestinian Arabic dialect,” Language Resources and Evaluation, vol. 51, no. 3, pp. 745-775, 2017. find
  23. Kohavi R., “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection,” in Proceedings of the 14 th international joint conference on Artificial intelligence, Montreal, pp. 1137-1143, 1995. find
  24. Korayem M., Aljadda K., and D. Crandall, “Sentiment/subjectivity analysis survey for languages other than English,” Social Network Analysis and Mining, vol. 6, no. 1, pp. 1-17, 2016. find
  25. Liu B., “Sentiment Analysis and Opinion Mining,” in Proceedings of Synthesis Lectures on Human Language Technologies, pp. 1-167, 2012. find
  26. McCallum A. and Nigam K., “A Comparison of Event Models for Naive Bayes Text Classification,” in Proceedings of AAAI-98 workshop on learning for text categorization, Madison, pp. 41-48, 1998. find
  27. Mountassir A., Benbrahim H., and Berraba I., “Sentiment Classification on Arabic Corpora. A Preliminary Cross-Study,” Document Numerique, vol. 16, no. 1, pp.73-96, 2013. find
  28. Petrillo M. and Baycroft J., “Introduction to Manual Annotation,” Fairview Research, pp 1-7, 2010, find
  29. Pustejovsky J. and A. Stubbs, Natural Language Annotation for Machine Learning, O’Reilly Media, 2012. find
  30. Rahab H., Zitouni A., and Djoudi M., “ARAACOM: ARAbic Algerian Corpus for Opinion Mining,” in Proceedings of the International Conference on Computing for Engineering and Sciences, Istanbul, pp. 35-39, 2017. find
  31. Refaeilzadeh P., Tang L., and Liu H., “Cross-Validation,” in Encyclopedia of Database Systems, Boston, pp. 532–538, 2009. find
  32. Rushdi-Saleh M., Martín-Valdivia M., Ureña López L., and Perea-Ortega J., “OCA: Opinion Corpus for Arabic,” Journal of the American Society for Information Science and Technology, vol. 62, no. 10, pp. 2045-2054, 2011. find
  33. Salzberg S., “On Comparing Classifiers : Pitfalls to Avoid and a Recommended Approach,” Data Mining and Knowledge Discovery, vol. 1, pp. 317-328, 1997. find
  34. Tunga G., “Part-of-Speech Tagging,” in Handbook of Natural Language Processing, Second Edi., CRC Press, pp. 205–235, 2010. find
  35. Vinodhini G. and Chandrasekaran R., “Sentiment Analysis and Opinion Mining : A Survey,” International Journal of Advanced Research in Computer Science and Software Engineering , vol. 2, no. 6, pp. 282-292, 2012. find
  36. Wang X., An K., Tang L., and Chen X.,“Short Term Prediction of Freeway Exiting Volume Based on SVM and KNN,” International Journal of Transportation Science and Technology, vol. 4, no. 3, pp. 337-352, 2015. find
  37. Zaidan O. and Callison-burch C., “The Arabic Online Commentary Dataset : an Annotated Dataset of Informal Arabic with High Dialectal Content,” in Proceedings of the 49 th Annual Meeting of the Association for Computational Linguistics, Portland, pp. 37-41, 2011. Hichem Rahab is currently working as an Assistant Professor in department of Mathematics and computer science in the University of Khenchela, Algeria. He obtained his master’s degree in computer science from Batna University, Algeria, 2012. His research interest includes machine learning, Arabic opinion mining and sentiment analysis. Abdelhafid Zitouni received his PhD in computer science in 2008 from the University of Constantine, Algeria. He is currently working as Professor in University of Constantine 2 Abdelhamid Mehri. His research interests include Cloud Computing, Security, and Arabic text mining field. Pr. Abdelhafid Zitouni has published many articles in International Journals and Conferences. He peer-reviewed conference and journal papers in the above research topics. Mahieddine Djoudi received a PhD in Computer Science from the University of Nancy, France, in 1991. His PhD thesis research was in Acoustic Phonetic Decoding for Standard Arabic Speech Recognition. He is currently working at Computer Science Department, Faculty of Fundamental and Applied Sciences at the University of Poitiers, France and member of TechNE Technology Enhanced Learning Research Laboratory. His main scientific interests are e-Learning, Mobile Learning, Cloud Computing, Information Literacy and Learning Analytics. He has published over 100 scientific papers. He is also a member of program committees, editor or reviewer for international journals or conferences proceedings. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahab-2021-enhanced-corpus-arabic-newspapers-comments/