📄 Sarcasm and Sentiment Detection In Arabic Tweets Using BERT-based Models and Data Augmentation

Summary

The paper describes a system for the shared task on Arabic sarcasm detection (Subtask 1) and three-way sentiment analysis (Subtask 2) on the ArSarcasm-v2 tweet dataset. The training data is small and skewed: only about one tweet in six is sarcastic, and positive tweets are similarly rare. After trying classical classifiers on tf-idf features and several deep networks, the authors fine-tuned seven Arabic and multilingual BERT-based models. MARBERT and QARiB performed best. To address the imbalance, they added extra training data from the ASAD sentiment corpus. Because almost all sarcastic tweets in the training set were also negative, they relabelled ASAD negative tweets as sarcastic and positive ones as not sarcastic. On the validation split, this raised F-scores by roughly 15 points for both subtasks. On the official test set, however, the gains largely disappeared, which suggests overfitting or a mismatch between the proxy labels and true sarcasm. An error analysis points to annotation noise, missing context, literal mentions of the word "sarcasm", and unprocessed emojis as the main failure sources.

Key points

• ArSarcasm-v2 has 12,548 training tweets and 3,000 test tweets. Sarcasm labels split 2,168 TRUE against 10,380 FALSE. Sentiment labels split NEU 5,747, NEG 4,621 and POS 2,180.
• Seven models were compared under identical settings: MARBERT, ArBERT, QARiB, AraBERTv02, GigaBERTv3, Arabic BERT and mBERT. All used 5 epochs, a learning rate of 2e-06, batch size 32 and an 80/20 train/validation split.
• On the original data, MARBERT reached 0.65 F1 on the sarcastic class, QARiB 0.60 and mBERT only 0.41. For sentiment (F-PN), QARiB scored 0.73 and MARBERT 0.71, but QARiB overfit more.
• Augmentation drew on 29,924 retrieved ASAD tweets and added 4,739 TRUE and 4,930 FALSE examples, giving 22,217 training tweets. On validation, MARBERT rose to 0.80 F1-sarcastic (QARiB to 0.75) and to 0.86 F-PN.
• MARBERT's official test scores were much lower than its validation scores: 0.52 F1-sarcastic and 0.71 F-PN.

Interesting ideas

• I1. Using a strong co-occurrence between labels (89% of sarcastic tweets are negative) to borrow examples from a neighbouring task's corpus is a cheap way to rebalance rare classes, though the test drop shows the proxy is noisy. — For Mova (learn): Silver irony or rare-frame examples could be harvested from correlated annotations, but only if they pass the same held-out negative-control check that exposed this paper's gap.
• I2. The large gap between validation and test scores is itself the key finding, because improvements measured on a split drawn from augmented or same-distribution data can be illusory. — For Mova (learn): Any LLM-teacher or silver-data gain should be confirmed on an untouched, differently sourced test set before it is accepted.
• I3. The error analysis found tweets that merely mention the word for sarcasm without being sarcastic, which separates talking about a phenomenon from performing it. — For Mova (frames): The irony module should treat words like "sarcasm" or "ironic" as topic mentions that the tree and frames must confirm, never as direct triggers.
• I4. Emojis that carry the opposite polarity to the literal text misled the classifier, yet such a clash between text and emoji is exactly the incongruity that often signals irony. — For Mova (frames): A polarity clash between an emoji and the evaluated frame could be logged as an explainable irony cue rather than averaged away.
• I5. The preprocessing splits hashtags into words by replacing underscores, collapses elongated characters, and normalises letter variants, turning noisy social tokens into analysable words. — For Mova (tokens): The en tokenizer could split hashtags such as #worst_day_ever and collapse repeated letters ("sooooo") before tagging, recording the original form for explanation.