Elamur Research⌕
☰
home / papers / 2021 / Preprocessing Solutions for Detection of Sarcasm and Sentiment for Arabic

Preprocessing Solutions for Detection of Sarcasm and Sentiment for Arabic

Summary

This paper presents a system from CRSTDLA (Algiers) for the Ar-Sarcasm 2021 shared task, which covers two tasks on Arabic tweets: detecting sarcasm and classifying sentiment. The authors argue that careful text preparation is the main lever, so they test a long list of cleaning and morphological steps. The cleaning steps include removing punctuation, emojis, stop words, diacritics, Latin script and repeated characters, plus normalising Arabic letters. The morphological steps are lemmatization, stemming and POS tagging. Each step is tried alone and then in cumulative stacks on ArSarcasm-v2. Two classifiers are used: a TF-IDF-based LSVC and a Keras BiLSTM. Development scores are very high, close to 98% for LSVC. On the official test set, however, the submitted system reaches only 33.71% F1-sarcastic and about 57.8% F1-PN for sentiment, below the participant averages. The study is a useful record of which surface normalisations help or hurt simple classifiers. Its sharp dev/test gap also warns against trusting a development set that may not match the test distribution.

Key points

  • Data. ArSarcasm-v2 relabels earlier sentiment corpora (SemEval 2017, ASTD) with sarcasm and dialect labels. It has 12,548 training tweets and 3,000 test tweets, and the authors used a 2,110-tweet dev set taken from the original ArSarcasm test split.
  • Morphological tools. Lemmatization uses WorADRetLemmatizer, stemming uses the ISRI Arabic Stemmer, and POS tags come from the NLTK PosTagger. Features for LSVC are TF-IDF with sklearn default settings.
  • BiLSTM setup.
    • Embedding layer: 20k-word vocabulary, 50-dimensional vectors, inputs truncated to 70 tokens.
    • Layers: a 512-unit BiLSTM, max pooling, a 256-unit dense layer and 0.4 dropout.
    • Training: RMSprop, batch size 128, 5 epochs.
  • Dev ablations.
    • Adding POS tags badly damaged the BiLSTM (sentiment 52.77, sarcasm 32.38).
    • Stacking every step pulled LSVC down to 95.42 for sentiment and 91.22 for sarcasm.
    • Letter normalisation (NA) was among the best single steps.
  • Test results.
    • Sarcasm: F1 33.71% and accuracy 72.87%, against an average of 52.52% and 80.48%.
    • Sentiment: F1-PN 57.87% in Table 4 (the abstract reports 57.80%) and accuracy 59.23%, against an average of 65.43% and 64.61%.

Interesting ideas

  • I1. Each preprocessing step is first measured on its own and then added one at a time in a cumulative chain, which exposes steps that look harmless alone but hurt when combined. — For Mova (tokens): Run the same single-step and cumulative ablation over the normalisation options in crate en, so that every rule in the tokenizer and lemmatizer has a measured effect.
  • I2. Piling on more cleaning did not steadily improve results; some additions, such as appending POS tags to tokens, made the BiLSTM much worse, probably by fragmenting its vocabulary. — For Mova (tree): When enriching parser inputs with tags or lemmas, check whether the richer token form splits rare words into even sparser units, and compare against plain forms.
  • I3. Development scores near the ceiling sat beside weak test scores, a gap that signals a dev split which does not represent the evaluation data. — For Mova (learn): Treat suspiciously high dev scores as a failure flag, and always evaluate on a separately sourced held-out set alongside the negative controls.
  • I4. The sarcasm corpus was created by adding a new label layer to existing sentiment corpora instead of collecting fresh text. — For Mova (frames): Add irony or sarcasm labels to texts Mova already analyses, so the irony module in crate prag can be tested against the sentiment-bearing frames on the same sentences.
  • I5. The maximum sequence length was set from corpus length statistics, cutting at 70 tokens although the longest training tweet has 148. — For Mova (learn): Size GPU batch and truncation limits for gparse from treebank sentence-length distributions, and record how many tokens are lost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

4 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lichouri2021preprocessing,
  title = {{Preprocessing Solutions for Detection of Sarcasm and Sentiment for Arabic}},
  author = {Lichouri, Mohamed and Abbas, Mourad and Benaziz, Besma and Zitouni, Aicha and Lounnas, Khaled},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.49/}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  2. Houda Bouamor, Wajdi Zaghouani, Mona Diab, Ossama Obeid, Kemal Oflazer, Mahmoud Ghoneim, and Abdelati Hawwari. 2015. A pilot study on arabic multi-genre corpus diacritization. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 80–88. find
  3. Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39. find our card
  4. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. ↗
  5. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1–22. find our card
  6. Jihen Karoui, Farah Banamara Zitoune, and Veronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. ↗
  7. Mohamed Lichouri and Mourad Abbas. 2020. Speech-trans@ smm4h’20: Impact of preprocessing and ngrams on automatic classification of tweets that mention medications. In Proceedings of the Fifth Social Media Mining for Health Applications Workshop & Shared Task, pages 118–120. find
  8. Mohamed Lichouri, Mourad Abbas, and Besma Benaziz. 2020. Profiling fake news spreaders on twitter based on tfidf features and morphological process. find
  9. Mohamed Maamouri, Ann Bies, Seth Kulick, Wajdi Zaghouani, David Graff, and Michael Ciul. 2010. From speech to trees: Applying treebank annotation to arabic broadcast news. In LREC. Citeseer. find
  10. Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1202–1210. find
  11. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2515–2519. find our card
  12. Ali Bou Nassif, Ashraf Elnagar, Ismail Shahin, and Safaa Henno. 2020. Deep learning for arabic subjective sentiment analysis: Challenges and research opportunities. Applied Soft Computing, page 106836. find
  13. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  14. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502– 518. find our card
  15. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. ↗
  16. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674. ↗
  17. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lichouri-2021-preprocessing-solutions-detection-sarcasm-sentiment/