Elamur Research⌕
☰
home / papers / 2021 / Machine Learning-Based Model for Sentiment and Sarcasm Detection

Machine Learning-Based Model for Sentiment and Sarcasm Detection

Summary

The paper addresses two related tasks on Arabic tweets: deciding whether a tweet is sarcastic, and labelling its sentiment as positive, negative, or neutral. Sarcasm is hard because the literal wording differs from what the writer means. Arabic adds difficulty through rich morphology and many dialects alongside Modern Standard Arabic. The work was entered in a shared competition covering Arabic sentiment and sarcasm identification, using the ArSarcasm-v2 dataset. The pipeline has three stages. First, preprocessing re-encodes the data as UTF-8 and strips hashtags, emojis, numbers, and URLs. Second, unigram/bigram features are weighted with TF/IDF. Third, a document-level classifier is applied. In five-fold cross-validation, an SVM beat Complement Naïve Bayes, Naïve Bayes, and Linear Regression on both tasks, so it was chosen for the blind test. Official test scores were far lower than the development accuracies, and the authors call the results moderate. The paper matters mainly as a simple, reproducible classical baseline for Arabic sarcasm and sentiment. It also shows how optimistic accuracy-based development numbers can be on imbalanced data.

Key points

  • Data: ArSarcasm-v2 contains 12,548 labelled training tweets and a blind test set of 3,000 tweets. Each tweet also carries dialect information.
  • The training set is strongly imbalanced:
    • Sarcasm: only 2,168 tweets (17.28%) are sarcastic.
    • Sentiment: 17.37% positive, 36.83% negative, and 45.80% neutral.
  • Five-fold CV results (sentiment / sarcasm):
    • SVM: 85.55% / 84.22%
    • CNB: 83.47% / 82.97%
    • NB: 78.92% / 78.10%
    • LR: 65.89% / 64.03%
    • SGD was also tried, but no score is shown.
  • Official metrics differ by task. Sarcasm is scored by F1 of the sarcastic class (F1-sarcastic). Sentiment is scored by the macro F-score over the positive and negative classes only (F-PN).
  • Blind-test submissions scored accuracies of 0.7460 and 0.5980, and official scores of 0.5936 and 0.2440. The extracted table is garbled, so which score belongs to which task is unclear from this copy.

Interesting ideas

  • I1. Score a skewed task by the F1 of the rare class you care about, not overall accuracy; this exposes systems that mostly predict the majority label. — For Mova (learn): Mova's evaluation of rare phenomena (irony, negation, unusual frames) should report F1 on the rare class itself so frequent easy cases cannot mask failures.
  • I2. The sentiment metric averages only over the positive and negative classes and ignores neutral, so a model gains nothing by defaulting to "neutral." — For Mova (answer): QA and test scoring could likewise exclude an "unknown/no answer" fallback from the main score, so readers are rewarded for committing to correct answers.
  • I3. The large drop from cross-validation accuracy to blind-test official scores shows that development numbers on the same distribution can badly overstate quality. — For Mova (learn): Mova should always keep a held-out blind set and a trivial-baseline negative control, and report the gap between them alongside development results.
  • I4. Sarcasm is framed as cases where the literal content and the intended meaning diverge. — For Mova (frames): crate prag could flag irony when the literal frame's evaluation clashes with world or context expectations, giving an explainable signal rather than an n-gram guess.
  • I5. A TF/IDF n-gram SVM is a cheap, transparent bag-of-words reference that several classifiers were benchmarked against. — For Mova (learn): such a model is useful as a negative-control baseline that Mova's structured, tree-and-frame approach to irony and sentiment must clearly beat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nayel2021machine,
  title = {{Machine Learning-Based Model for Sentiment and Sarcasm Detection}},
  author = {Nayel, Hamada and Amer, Eslam and Allam, Aya and Abdallah, Hanya},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.51/}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591, Portland, Oregon, USA. Association for Computational Linguistics. find
  2. Hamada A. Nayel and Shashirekha H. L. 2019. DEEP at HASOC2019: A machine learning framework for hate speech and offensive language detection. In Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pages 336–343. CEUR-WS.org. find
  3. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find our card
  4. Hamada A. Nayel, Walaa Medhat, and Metwally Rashad. 2019. BENHA@IDAT: Improving Irony Detection in Arabic Tweets using Ensemble Approach. In Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pages 401–408. CEUR-WS.org. find
  5. D. Al-Ghadhban, E. Alnkhilan, L. Tatwany, and M. Al-razgan. 2017. Arabic sarcasm detection in twitter. In 2017 International Conference on Engineering MIS (ICEMIS), pages 1–7. find
  6. Sergios Theodoridis and Konstantinos Koutroumbas. 2009. Chapter 3 - Linear Classifiers. In Pattern Recognition (Fourth Edition), fourth edition edition, pages 91 – 150. Academic Press, Boston. find
  7. Ahmed Alsayat and Nouh Elmitwally. 2020. A comprehensive study for arabic sentiment analysis (challenges and applications). Egyptian Informatics Journal, 21(1):7–12. find
  8. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. Language in Mind: A Tribute to Neil Smith on the Occasion of his Retirement. find
  9. Naaima Boudad, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb. 2018. Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal, 9(4):2479–2490. find
  10. Jihen Karoui, Farah Banamara Zitoune, and Véronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. Arabic Computational Linguistics. find
  11. E. Lunando and A. Purwarianti. 2013. Indonesian social media sentiment analysis with sarcasm detection. In 2013 International Conference on Advanced Computer Science and Information Systems (ICAC-SIS), pages 195–198. find
  12. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  13. Hamada Nayel. 2020. NAYEL at SemEval-2020 task 12: TF/IDF-based approach for automatic offensive language detection in Arabic tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2086–2089, Barcelona (online). International Committee for Computational Linguistics. find
  14. Hamada A. Nayel. 2019. NAYEL@APDA: Machine Learning Approach for Author Profiling and Deception Detection in Arabic Texts. In Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pages 92–99. CEUR-WS.org. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nayel-2021-machine-learning-based-model-sentiment/