📄 iCompass at Shared Task on Sarcasm and Sentiment Detection in Arabic

Summary

The paper describes the iCompass team's entry to the 2021 Arabic shared task on sarcasm detection (Subtask 1) and three-way sentiment classification (Subtask 2). The task is hard for three reasons: labelled data is scarce, sarcasm depends on culture, and the same word can carry opposite polarity in different Arabic dialects. The team used the ArSarcasm-v2 tweet corpus, split it 80:20 with a deliberately balanced validation part, and stripped emojis, URLs, diacritics and punctuation. They then fine-tuned four pretrained encoders: mBERT, AraBERT, ARBERT and MARBERT. MARBERT, pretrained on dialect-rich tweets, was clearly best on both subtasks, and it was used for the final submission. On the official test set the system placed 8th in sentiment analysis but only 21st in sarcasm detection. The authors trace the weak sarcasm result mainly to class imbalance in the training data. The work shows that matching the pretraining domain and dialect matters more than generic multilinguality for noisy, dialectal social-media text.

Key points

• ArSarcasm-v2 contains 12548 tweets, each labelled for sentiment (POS/NEG/NEU), sarcasm (TRUE/FALSE) and dialect (msa, gulf, egypt, levant, magreb). After the split, training kept only 913 sarcastic tweets against 9125 non-sarcastic ones.
• MARBERT was pretrained on 128 GB of Arabic tweets with a 100k WordPiece vocabulary. The final setup used a learning rate of 2e-5, batch size 128 and max sequence length 128, with 8 epochs for sarcasm and 3 for sentiment. Each run took about 5 minutes on a Colab TPU.
• On development data for sarcasm, MARBERT reached 68.8% accuracy and 57.7% F1-sarcastic, against 47% for AraBERT and 31.2% for mBERT. Sarcastic-class precision was high (89.7%) but recall was low (42.5%).
• On development data for sentiment, MARBERT reached 72% accuracy and 72.95% F-PN, against 68.7% for AraBERT and 60.15% for mBERT. Neutral was the class most often confused with the others.
• Official test results were 70.85% F-PN (rank 8; the winner scored 74.80%) and 48.6% F1-sarcastic (rank 21; the winner scored 62.25%).

Interesting ideas

• I1. The authors show that one word can signal praise in one dialect and insult in another, so polarity belongs to a word-and-variety pair rather than to the word alone. This matters for any system that handles regional varieties. — For Mova (frames): The irony component could store evaluative polarity with a register or variety tag, so a reversal is flagged only when the speaker's variety makes the surface word positive.
• I2. The paper uses an operational definition of sarcasm: an implicitly negative stance expressed through positive surface wording. It also notes cases where praise at the start of a tweet flips to criticism at the end. This gives a testable structural cue instead of a vague label. — For Mova (frames): Implicature detection could compare the polarity of the explicit evaluation with the polarity implied by the described situation, and mark a mismatch as candidate irony with an explanation trace.
• I3. The comparison across encoders suggests that pretraining on text resembling the target domain (dialectal tweets) beats both general multilingual models and larger formal-text models. — For Mova (learn): When choosing or training witness parsers for silver data, give extra weight to the parser whose training text best matches the target genre, and track agreement by genre.
• I4. Reporting per-class precision and recall, plus confusion matrices, exposed a model that is very precise on sarcasm but misses most of it. A single accuracy figure would hide this failure. — For Mova (learn): Evaluation reports could always break results down by class or phenomenon, so a high-precision, low-recall pattern in rare cases like irony is caught early.
• I5. The split design is a cautionary example: balancing the validation set pulled most of the rare class out of training, and the authors themselves name the resulting imbalance as the main cause of weak sarcasm scores. — For Mova (learn): Held-out sets for rare phenomena should be built without starving the training pool, for example with stratified sampling, and the training-side class counts should be checked before blaming the model.