📄 SarcasmDet at Sarcasm Detection Task 2021 in Arabic using AraBERT Pretrained Model

Summary

The paper presents SarcasmDet, a system built for sub-task 1 (Sarcasm Detection) in WANLP 2021's shared task covering sarcasm and sentiment detection in Arabic. The task is to decide whether an Arabic tweet is sarcastic. The authors used the ArSarcasm-v2 dataset, which is strongly skewed toward non-sarcastic tweets. They applied light cleaning with NLTK and then fine-tuned several pretrained transformers through HuggingFace and Simple Transformers: XLM-R, mBERT (cased and uncased), AraBERT-v01 and AraBERT-v02. The Arabic-specific AraBERT-v02 was the strongest single model. Their final submission combined both AraBERT versions with hard voting. This ensemble placed fifth of 27 teams on the official metric, F1 for the sarcastic class. It also had the best accuracy of any entry in the competition. The work shows that a monolingual Arabic encoder clearly beats multilingual ones on dialectal, figurative social-media text. It also shows that a simple voting ensemble gives a measurable gain on an imbalanced, hard pragmatic task.

Key points

• Data: ArSarcasm-v2 has 12,548 training tweets with sarcasm, sentiment and dialect labels, plus 3,000 test tweets. The training set holds 2,168 sarcastic and 10,380 non-sarcastic tweets.
• AraBERT versions:
  • AraBERT-v01 was pretrained on 77M sentences (23GB, 2.7B words).
  • AraBERT-v02 was pretrained on 200M sentences (77GB, 8.6B words).
• Single models (F1-sarcastic, development phase):
  • AraBERTv02: 0.5650 (learning rate 1e-5, seed 17, batch size 16, 5 epochs)
  • AraBERTv01: 0.5459
  • mBERT-cased: 0.4811
  • mBERT-uncased: 0.4374
  • XLM-R: 0.3993
• Official test result (hard-voting ensemble of AraBERT-v01 and v02):
  • F1-sarcastic: 0.5989
  • Accuracy: 0.7830, the highest in the task
  • Macro F1: 0.7251
  • Precision: 0.7268
  • Recall: 0.7235
• Training details: Preprocessing normalized repeated characters and removed emojis, links and HTML tags. Lowering XLM-R's learning rate from its default to 1e-5 improved its results.

Interesting ideas

• I1. The team ranked first on accuracy but only fifth on minority-class F1, which shows how a skewed label distribution makes plain accuracy look better than real detection of the rare class. — For Mova (learn): Irony, negation or rare frame evaluations should report per-class F1 for the rare label next to overall accuracy, so a reader that never flags the rare case cannot look good.
• I2. Hard voting over two variants of the same model family, trained on different amounts of pretraining data, gave a clear gain over the best single variant at almost no engineering cost. — For Mova (learn): Silver-data agreement across witness parsers could also mix differently trained versions of one parser, since they disagree in useful ways while sharing a label scheme.
• I3. The authors compared each model with and without cleaning, rather than assuming normalization always helps. Their ablation table shows the effect is not uniform. — For Mova (tokens): Normalizing elongated words like "soooo", or stripping emojis and links, should be tested as a toggle with an ablation, because such marks may carry pragmatic signal worth keeping.
• I4. A monolingual encoder trained on far more in-language text beat multilingual models by a wide margin on dialectal, figurative tweets. Pretraining data that matches the target domain mattered more than broad language coverage. — For Mova (frames): An irony detector in prag should be trained or calibrated on informal, in-domain English, and its verdict should be kept as an explicit, explainable speech-act feature rather than a hidden score.
• I5. Using one shared learning rate (1e-5) across all candidate models fixed XLM-R's poor default and made the model comparison fairer. — For Mova (tree): When comparing gparse encoder backbones, hyperparameters should be fixed and reported uniformly, so that differences in LAS reflect the models rather than their defaults.