Elamur Research⌕
☰
home / papers / 2012 / Preprocessing Egyptian Dialect Tweets for Sentiment Mining

Preprocessing Egyptian Dialect Tweets for Sentiment Mining

Paper · 2012 · Amira Shoukry and Ahmed Rafea

Summary

Arabic sentiment analysis has few resources, such as labeled corpora and polarity lexica, and dialectal social media text is especially noisy. The authors measure how three preprocessing steps (normalization, stemming and stop-word removal) affect sentence-level polarity classification of Egyptian-dialect tweets. They collected and annotated 1000 tweets as positive or negative. They then compared two classification families:

  • Machine learning (ML): feature vectors fed to an SVM.
  • Semantic orientation (SO): sentiment-word lists whose matches are summed to decide polarity.

They reused an existing Ruby normalizer and built their own dialect-aware light stemmer. The stemmer uses Egyptian prefix and suffix sets, rules for broken plurals, and exception lists, and it checks each intermediate result against a dictionary. They tested each step cumulatively. Preprocessing raised ML performance by about 4.5% and SO measures by 2–7%. The best accuracy was 78.8% for ML and 75.9% for SO. The custom stemmer only slightly beat a generic light stemmer. The work shows that cheap, dialect-aware normalization matters for noisy, low-resource text.

Key points

  • The data set is 1000 Egyptian-dialect tweets labeled positive or negative. It was collected with an annotation tool built together with LINK-Development under an ITIDA-sponsored project.
  • Normalization rules cover:
    • diacritics (Tashkeel) and elongation (Tatweel);
    • Hamza, Alef and lam-alef variants;
    • yeh/alef-maqsura and heh/teh-marbuta.
  • The stemmer runs prefix removal, then suffix removal, then infix (broken-plural) handling. It extends rules from Beltagy and Rafea (2011), which builds on light10.
  • For SO, accuracy rose from 0.689 on raw tweets to 0.759 on stemmed tweets. The custom stemmer beat the generic light stemmer by about 0.5% (SO) and 1% (ML).
  • Stop-word removal added only about 0.1%. Positive tweets reached near-best results after normalization alone, which suggests they are less noisy than negative ones.

Interesting ideas

  • I1. Before any affix rule fires, the word is looked up in lists of irregular forms: words that only look affixed, and irregular plurals mapped to their singulars. A word found there bypasses the rules entirely.
  • I2. After each stripping stage, the candidate form is checked against a dictionary, and stemming stops if the next cut would leave unattested territory. This guards against over-stemming without needing a full morphological analyzer.
  • I3. The same normalization and stemming were applied both to the input tweets and to the stored sentiment-word lists, so matching happens between forms in one shared space. The authors note that a leftover mark in one side broke matches.
  • I4. An MSA stemmer wrongly treated a final "-aan" in the Egyptian word for "because" as a dual suffix. The authors cite this as a reason to build dialect-specific affix rules rather than reuse standard-language ones.
  • I5. The authors explain the lexicon method's shortfall with three failure classes: the tweet has no sentiment word; the word is missing from the list; or the word survives preprocessing in a variant form.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en lemmatizer could consult an explicit exception table (e.g. "news", "species", irregular plurals) before suffix stripping, so every lemma decision has a traceable reason.
  • I2 (tokens): Candidate lemmas from suffix rules could be validated against the WordNet/lexicon vocabulary at each step, and that validation could be logged as part of the explanation.
  • I3 (world): When linking text words to compiled knowledge (verb classes, ontology entries), both sides should pass through the identical normalizer, with mismatch cases surfaced as diagnostics.
  • I4 (tree): When converting or parsing dialectal or informal English, rules tuned on standard text should be audited for false triggers, and their applicability should be scoped per dialect.
  • I5 (learn): Evaluation reports could bucket errors into similar "no cue / cue not in lexicon / cue misnormalized" classes, pointing repair work at the right component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 972 works

Cite

Cite as Amira Shoukry and Ahmed Rafea. 2012. Preprocessing Egyptian Dialect Tweets for Sentiment Mining. Fourth Workshop on Computational Approaches to Arabic-Script-based Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shoukry2012preprocessing,
  title = {{Preprocessing Egyptian Dialect Tweets for Sentiment Mining}},
  author = {Amira Shoukry and Ahmed Rafea},
  booktitle = {Fourth Workshop on Computational Approaches to Arabic-Script-based Languages},
  year = {2012},
  url = {https://aclanthology.org/2012.amta-caas14.7/}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rehab Duwairi, Mohamed N. Al-Refai, and Natheer Khasawneh. 2009. Feature reduction techniques for Arabic text categorization. Journal of the American Society for Information Science and Technology, 60(11), 2347–2352. ↗
  2. Khaled F. Shaalan and Hafsa Raza. 2009. NERA: Named entity recognition for Arabic. Journal of the American Society for Information Science and Technology, 60(8), 1652–1663. ↗
  3. Sara Morsy and Ahmed Rafea. 2012. Improving Document-Level Sentiment Classification Using Contextual Valence Shifters Natural Language Processing and Information Systems Lecture Notes in Computer Science Volume 7337, 2012, pp 253-258 find
  4. Leah S. Larkey, Lisa Ballesteros, and Margaret E. Connell. 2007. Light stemming for Arabic information retrieval. In Arabic Computational Morphology, A. Soudi, A. van der Bosch, and G. Neumann, Eds. 221– 243. ↗
  5. S. Khoja, and R. Garside. 1999. Stemming Arabic text. Tech. rep. Computing Department, Lancaster University, Lancaster, U.K. find
  6. Kazem Taghva, Rania Elkhoury, and Jeffery Coombs. 2005. Arabic stemming without a root dictionary. ITCC 1, 152–157. find
  7. Samhaa R. El-Beltagy and Ahmed Rafea. 2011. An accuracy-enhanced light stemmer for arabic text. ACM Trans. Speech Lang. Process. 7, 2, Article 2 (Feb. 2011), 22 pages. ↗
  8. Abdusalam Nwesri, S. M. M. Tahaghoghi, and Falk Scholer. 2005. Stemming Arabic conjunctions and prepositions. In Proceedings of the 12th International Symposium on String Processing and Information Retrieval (SPIRE’05). Lecture Notes in Computer Science, vol. 3772, Springer, 206–217. find
  9. Abduelbaset Goweder, Massimo Poesio, and Anne De Roeck. 2004. Broken plural detection for arabic information retrieval. In Proceedings of the Annual ACM Conference on Reseurch and Development in Information Retvieval (SIGIR’04). find
  10. Abduelbaset Goweder, Massimo Poesio, Anne De Roeck, and Jeff Reynolds. 2004. Identifying broken plurals in unvowelised Arabic text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  11. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update; SIGKDD Explorations, Volume 11, Issue 1. find
  12. Mohamed Elhawary, and Mohamed Elfeky. 2010. find
  13. Mining Arabic Business Reviews, Google Inc., Mountain View, CA, USA, 2010 IEEE International Conference on Data Mining Workshops. Bo Pang and Lillian Lee. Thumbs up? Sentiment Classification using Machine Learning Techniques, Department of Computer Science, Cornell University, Shivakumar Vaithyanathan, IBM Alma den Research Center. find
  14. Laila Khreisata. 2009. A machine learning approach for Arabic text classification using N-gram frequency statistics, Journal of Informatics, Vol 3, Issue 1, January 2009, Pages 72-77. find
  15. Eiman T. Al-Shammari. 2009. A Novel Algorithm for Normalizing Noisy Arabic Text, Computer Science and Information Engineering, WRI World Congress on , vol.4, no., pp.477-482, March 31 2009-April 2 2009 doi: 10.1109/CSIE.2009.952 ↗
  16. Amira Shoukry, and Ahmed Rafea. 2012. Sentence-level Arabic sentiment analysis, Collaboration Technologies and Systems (CTS), 2012 International Conference on , vol., no., pp.546-550, 21-25 May 2012 doi: 10.1109/CTS.2012.6261103 ↗
  17. Wentian Li. 1992. Random texts exhibit Zipf's-law-like word frequency distribution, Information Theory, IEEE Transactions on , vol.38, no.6, pp.1842-1845, Nov 1992 doi: 10.1109/18.165464 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shoukry-2012-preprocessing-egyptian-dialect-tweets-sentiment/