📄 UMUTeam at SemEval-2022 Task 6: Evaluating Transformers for detecting Sarcasm in English and Arabic

Summary

The paper describes UMUTeam's system for iSarcasm (SemEval-2022 Task 6). The team took part in two subtasks: binary sarcasm detection for English and Arabic, and multi-label recognition of ironic-speech mechanisms for English. The corpus was labelled by the tweet authors themselves and is strongly skewed toward non-sarcastic texts. The team built a separate pipeline for each language. They extracted four feature sets: language-independent stylometric and PoS features from UMUTextStats, fastText word embeddings, fastText sentence embeddings, and sentence embeddings from fine-tuned BERT or Arabic BERT. They trained a hyperparameter-searched neural network on each set and then combined the sets in two ways. The first was knowledge integration, which fuses features inside one network. The second was ensembling by vote or by probability averaging. Knowledge integration worked best on their own validation split. However, the official scores were weak: the F1 for the sarcastic class reached only 17.97 in English and 31.75 in Arabic. The Arabic validation scores were near-perfect but collapsed on the test set. The paper is useful mainly as a cautionary case about imbalanced data and validation splits that overestimate performance.

Key points

• Corpus sizes were 4868 English and 4900 Arabic texts. The ratio of sarcastic to non-sarcastic texts was about 1:3 in English and 1:6 in Arabic.
• In subtask 2 the six labels were very unevenly represented, from 713 sarcasm examples down to 10 understatement examples. Each label was handled as its own binary classifier.
• On the English validation split, BERT features alone reached a sarcastic-class F1 of 45.10. Knowledge integration raised this to 45.82, while the vote and averaging ensembles fell to about 38.
• Arabic validation macro-F1 was around 93–97 for every feature set. On the official test set this dropped sharply, for example from 94.07 to 33.82 with the LF features, with a best of 44.85 using knowledge integration.
• Official ranks were 41st in English (the top system scored 60.52) and 22nd of 32 in Arabic (the top system scored 56.32). The subtask 2 submission was not recorded because of an unexplained error.

Interesting ideas

• I1. The authors chose large batch sizes (128–512) on purpose, so that every batch of a heavily imbalanced dataset contains enough minority-class examples to give a useful gradient.
• I2. Fusing all feature sets inside a single network (knowledge integration) beat voting or averaging separately trained models, while the probability-averaging ensemble nearly failed on rare traits.
• I3. The Arabic validation scores were near-perfect, yet the test scores collapsed, and a duplicate check did not explain the gap. Validation numbers that look too good should be treated as a warning sign, not a success.
• I4. The model labels figurative-speech mechanisms (overstatement, understatement, rhetorical question, satire) as separate, independent yes/no decisions rather than treating sarcasm as a single category.
• I5. The paper keeps interpretable, language-independent stylometric and PoS features (TTR, punctuation, length) as a separate baseline next to transformer embeddings, which shows how much the transparent cues contribute on their own.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When training classifiers on rare phenomena such as irony or rare dependency relations, batch size or sampling can be set so each batch carries rare examples, which should give a more stable signal.
• I2 (frames): An irony detector in prag could feed structural cues and embedding cues into one decision layer instead of averaging separate detectors, and still log which cue drove each decision.
• I3 (learn): Evaluation could add automatic alarms for implausibly high held-out scores, and run negative controls such as shuffled labels and checks for author or topic overlap before trusting any split.
• I4 (frames): The irony component could output a set of mechanism flags, each with its own evidence, so that implicature reasoning knows which kind of non-literal meaning applies.
• I5 (frames): Cheap tagger-level cues could serve as an explainable baseline for irony detection, measured against heavier models to show what structure alone can capture.