📄 A Contextual Word Embedding for Arabic Sarcasm Detection with Random Forests

Summary

This shared-task paper tackles sarcasm detection in Arabic tweets. Arabic suffers from small and skewed datasets, weak preprocessing tools, and strong dialect variation. The authors train on ArSarcasm-V2 (12548 tweets, about 17% sarcastic) and balance the classes by duplicating sarcastic tweets and dropping some non-sarcastic ones. They clean the tweets with regular expressions and replace emojis with Arabic names taken from a translated Kaggle emoji list. Each tweet then goes through AraBERT-base to obtain contextual token vectors. Stop words and affix sub-tokens are removed only after encoding, and the remaining vectors are averaged into a 768-dimensional tweet vector. A default scikit-learn random forest classifies these vectors.

On the official unseen test set the system reached 0.5189 F1-sarcastic and 0.7533 accuracy, clearly above a TF-IDF variant (0.41 F1-sarcastic). The authors also admit that their internal 0.93 score was inflated, because they augmented the data before splitting it. The paper is mainly useful as a simple contextual-embedding baseline and as a cautionary note on evaluation leakage.

Key points

• Data: ArSarcasm-V2 with sarcasm, sentiment (POS/NEG/NEU) and dialect labels (MSA, Gulf, Egypt, Levant, Magreb). The 10380 non-sarcastic vs 2168 sarcastic tweets were rebalanced to 6018 per class, 12036 in total.
• Representation: AraBERT base configuration (L=12, H=768, A=12, 110M parameters). Token embeddings are mean-pooled into a 768-feature sentence vector.
• Classifier: a random forest with default settings (100 estimators, unlimited depth, Gini criterion). Tweets left empty after cleaning are labelled non-sarcastic by default.
• Official results: F1-sarcastic 0.5189, accuracy 0.7533, macro-F1 0.6765, precision 0.6858, recall 0.6700.
• TF-IDF comparison: F1-sarcastic 0.41, accuracy 0.73, macro-F1 0.62, recall only 0.34.
• The internal 80/20 validation gave 0.93 F1 for both classes. This was an artefact of duplicated tweets leaking into the validation split.

Interesting ideas

• I1. Encode the full, untouched text first, then filter stop words and affixes from the contextual output, so that small function words still shape the context of their neighbours. — For Mova (tokens): keep every token, including negators and particles, through tagging and lemmatization, and apply any filtering only to downstream views, never to the input of contextual models.
• I2. The WordPiece tokenizer's prefix/suffix markers are reused as a cheap stemmer for dialect forms that no rule-based stemmer covers. — For Mova (tokens): subword segmentation from gparse's tokenizer could suggest candidate affix splits for unknown or nonstandard English words, to be checked against the morphology rules.
• I3. Oversampling by duplication before the train/validation split leaks copies into validation and makes a weak model look excellent. The paper shows this failure openly. — For Mova (learn): always split by source item before any augmentation or silver-data expansion, and add a duplicate-overlap check as a standing negative control in evaluation.
• I4. Emojis are replaced by their names in the target language, turning opaque symbols into words the encoder understands. — For Mova (tokens): map emoji and emoticons to lexical tokens with lemmas, so later stages can treat them as evaluative words rather than noise.
• I5. Sarcasm is treated as a labelled property of the whole utterance and scored on the sarcastic class specifically, not on overall accuracy. — For Mova (frames): when evaluating the irony component in prag, report F1 on the ironic class separately, since accuracy hides poor recall on a rare phenomenon.