📄 akaBERT at SemEval-2022 Task 6: An Ensemble Transformer-based Model for Arabic Sarcasm Detection

Summary

The paper describes a system for SemEval-2022 Task 6 (iSarcasmEval), Sub-task A on the Arabic data, where each tweet is labelled sarcastic or non-sarcastic. Arabic sarcasm is hard to detect for several reasons: the language is morphologically rich, dialects vary widely, resources are scarce, and tweets are short, noisy and full of slang. The training set has 3,102 tweets and is skewed, with 745 sarcastic against 2,357 non-sarcastic.

The authors fine-tuned three Arabic pretrained transformers: AraBERT, MARBERT and AraELECTRA. To fix the class imbalance, they created extra sarcastic examples with NLPAug, which inserts or substitutes words using AraBERT's contextual embeddings. A hard majority vote then combined the three models' predictions.

On their own 60/20/20 split, augmentation raised the sarcastic-class F1 from the mid-70s to roughly 87–91% for the single models, and the ensemble reached 93%. On the official hidden test set, however, the sarcastic F1 was only 0.4438, which placed the system sixth. The work shows that simple balancing and voting help on in-house data, but those gains did not carry over to the real test.

Key points

• Data: the Arabic iSarcasmEval training set has 3,102 tweets (2,357 non-sarcastic, 745 sarcastic) with dialect labels. The 1,400-tweet test set was released unlabelled.
• Augmentation: NLPAug, using AraBERT as the contextual embedding model, inserted or replaced words in sarcastic tweets until the two classes were equal in size.
• Results without augmentation: sarcastic-class F1 was 74% for AraBERT, 69% for AraELECTRA, 77% for MARBERT and 77% for the ensemble.
• Results with augmentation:
  • AraBERT 87%, AraELECTRA 90%, MARBERT 91% and ensemble 93%.
  • The abstract assigns the 90% and 91% scores to the models in a different order than the results section does.
• Official evaluation: the system ranked sixth, with F1 of 0.4438 for sarcastic and 0.6222 for non-sarcastic tweets.
• Pretraining corpora: MARBERT was pretrained on 128 GB of text, half of it tweets. AraBERT was pretrained on 66 GB of news, and the authors credit MARBERT's tweet-heavy data for its better handling of dialects.

Interesting ideas

• I1. Three models with different pretraining are combined by a plain majority vote, so no single model's blind spots decide the label, and the method needs no extra training. — For Mova (learn): the same rule could build silver data from several witness parsers, keeping a head/label decision only when most parsers agree and sending split decisions to the LLM teacher tier.
• I2. The minority class is enlarged by having a contextual language model swap or insert words, which yields new examples that keep the label at almost no annotation cost. — For Mova (learn): this could create variants of rare constructions for training, provided each variant is re-parsed and dropped if its tree no longer matches the intended structure.
• I3. The in-house split gave 93%, while the official sarcastic F1 was 0.44. This gap is a warning that augmented, self-made splits can badly overstate quality, especially when it is unclear whether augmented copies reached the held-out part. — For Mova (learn): augment only the training portion, keep a held-out set that is never touched, and report negative controls so that inflated scores are caught before release.
• I4. The paper frames sarcasm as positive surface wording that hides a negative intent, as in praising bad weather. — For Mova (frames): the irony component in prag could explicitly compare the polarity of an utterance with the state of the text's world, and flag irony when they clash, which yields an explainable reason rather than a bare classifier score.
• I5. Matching the pretraining domain to the input genre matters: tweet-heavy MARBERT handled dialectal text best. — For Mova (tree): when training or choosing parsers for informal input, weight treebanks of web or social text more heavily, and record genre so that gparse can be checked separately on each domain.