Elamur Research⌕
☰
home / papers / 2022 / FII UAIC at SemEval-2022 Task 6: iSarcasmEval - Intended Sarcasm Detection in English and Arabic

FII UAIC at SemEval-2022 Task 6: iSarcasmEval - Intended Sarcasm Detection in English and Arabic

Summary

The authors (FII UAIC team) took part in SemEval-2022 Task 6, iSarcasmEval, and addressed subtask A: deciding whether an English tweet is sarcastic. They deliberately trained only on the 3468-tweet set supplied by the organisers, without outside corpora or augmentation. Tweets were lowercased, tokenized, stripped of non-alphanumeric tokens, lemmatized and filtered with the nltk stopword list. They were then turned into one-hot, doc-term or tf-idf matrices. On these features the team compared Bernoulli, Multinomial and Complement Naïve Bayes, logistic regression, and soft-margin SVMs with linear, polynomial and RBF kernels. They also built a small GRU network for the multi-label sarcasm categories. Multinomial NB with doc-term counts came out on top, while the SVMs did worst. However, every reported number was measured on the training data, since no test split was available when they worked, so overfitting cannot be excluded. The paper mainly shows how weak plain bag-of-words baselines are on a small, imbalanced corpus of intended sarcasm.

Key points

  • The dataset has 3468 tweets, each with 8 binary columns. "sarcastic" is the binary target, and the multi-class setting uses seven category labels: irony, satire, sarcasm, ambiguity, rhetorical questions, and both under- and overstatement. The labels are heavily skewed.
  • Bernoulli NB used one-hot features, while Multinomial and Complement NB used doc-term or tf-idf features. SVM C and gamma were tuned by randomized search with cross-validation over exponential distributions, and even after 200 iterations no good RBF model emerged.
  • With doc-term input, Multinomial NB showed a gap of about 8% between accuracy and balanced accuracy, and kept positive-class precision and recall above 80%. It also had the largest ROC area.
  • Logistic regression rarely predicted the sarcastic class (low recall), but its positive predictions were almost always right (near-100% precision).
  • The GRU model had a 7-way softmax output, 10% dropout, batch normalization after the input, and 5-dimensional word2vec embeddings. It performed poorly, which the authors attribute to too little data.

Interesting ideas

  • I1. Reporting balanced accuracy next to plain accuracy, and reading the gap between them, exposes how much a model profits from the majority class on skewed labels. — For Mova (learn): Every evaluation of rare-phenomenon detectors (irony, speech-act types) could print both figures, so that a large gap flags a detector that is coasting on the majority label.
  • I2. The paper warns, by its own example, that scoring on training data proves nothing. A naive model topping the table without a held-out split mostly measures memorisation. — For Mova (learn): A bag-of-words Naïve Bayes trained and scored on the same split could serve as a negative control, so that an explainable pipeline must beat it on held-out data to claim real understanding.
  • I3. Different classifiers landed at different points of the precision/recall trade-off. A model that seldom says "sarcastic" but is nearly always right when it does is valuable when false alarms are costly. — For Mova (frames): The irony component in crate prag could be tuned toward high precision, marking an utterance ironic only on strong evidence and otherwise keeping the literal reading for downstream stages.
  • I4. The authors note that features built from a closed training vocabulary break down when test texts bring unseen words, and that silently dropping them is only a stopgap. — For Mova (tokens): The lemmatizer and tagger in crate en should have explicit, logged out-of-vocabulary handling, so that unknown words still get a principled analysis instead of vanishing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.136

@inproceedings{manoleasa2022fii,
  title = {{FII UAIC at SemEval-2022 Task 6: iSarcasmEval - Intended Sarcasm Detection in English and Arabic}},
  author = {Manoleasa, Tudor and Gifu, Daniela and Sandu, Iustin},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.136},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.136}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amardeep Kumar, Vivek Anand. 2020. Transformers on Sarcasm Detection with Context. Proceedings of the Second Workshop on Figurative Language Processing, pages 88–92. find our card
  2. C. Van Hee, E. Lefever, and V. Hoste. 2018. Exploring the fine-grained analysis and automatic detection of irony on Twitter. Lang Resources Evaluation Vol. 52, pages 707–731. find
  3. Christopher Bishop. 2006. Pattern Recognition and Machine learning. Springer, New York. Complement scikit. Naive Bayes documentation of https://scikit-learn.org/ stable/modules/naive_bayes.html# complement-naive-bayes . ↗
  4. Dan Alexandru and Daniela Gı̂fu. 2020. Tracing humor in Edited News Headlines. Ludic, Co-design and Tools Supporting Smart Learning Ecosystems and Smart Education, pages 187–196. Springer Singapore. find
  5. Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR. find our card
  7. Gı̂fu, Daniela. 2010. The Discourse of the Written Press and the Violence of Symbols. PhD thesis, Faculty of Philosophy and Political Studies,“Alexandru Ioan Cuza” University of Iaşi. find
  8. Gifu Daniela, Samson Mihai. 2021. FII FUNNY at SemEval-2021 Task 7: HaHackathon: Detecting and rating Humor and Offense. Proceedings of the 15th International Workshop on Semantic Evaluation, (SemEval-2021), ACL, Bangkok, Thailand, pages 1226–1231, DOI: 10.18653/v1/2021.semeval-1.174. ↗
  9. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. Jason D. M. Rennie, Lawrence Shih, Jaime Teevan, David R. Karger. Tackling the Poor Assumptions of Naive Bayes Text Classifiers. Massachusetts Institute of Technology. John S. Leggitt and Raymond W. Gibbs Jr. find our card
  10. . 2000. Emotional reactions to verbal irony. find
  11. Reyes, Antonio and Rosso, Paolo and Buscaldi, Davide. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. find
  12. Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of the 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW). find
  13. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. AAAI. Tom M. Mitchell. http://www.cs.cmu.edu/˜tom/ mlbook/NBayesLogReg.pdf . ↗
  14. Tom M. Mitchell. 1997. Machine Learning,. McGraw Hill Education, New York. find
  15. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in between. ACL, pages 1010–1020. find our card
  16. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489. Precision Recall F1-Score Support Overall - 0.97 0.66 0.78 - balanced accuracy - - - 0.8254 accuracy - - - 0.9094 macro avg 0.93 0.83 0.86 - weighted avg 0.92 0.91 0.90 - *Table 1: Multinomial NB with doc-term encoding* Precision Recall F1-Score Support Overall 0.89 0.99 0.94 - 0.98 0.65 0.78 - balanced accuracy - - - 0.8208 accuracy - - - 0.9077 macro avg 0.93 0.82 0.86 - weighted avg 0.91 0.91 0.90 - *Table 2: Multinomial NB with one hot encoding* Precision Recall F1-Score Support Overall 0.76 1.00 0.87 - 1.00 0.07 0.13 - balanced accuracy - - - 0.5340 accuracy - - - 0.7670 macro avg 0.88 0.53 0.50 - weighted avg 0.82 0.77 0.68 - *Table 3: Multinomial NB with tf-idf encoding* Precision Recall F1-Score Support Overall 0.96 1.00 0.98 - 0.99 0.87 0.93 - balanced accuracy - - - 0.9323 accuracy - - - 0.9653 macro avg 0.98 0.93 0.95 - weighted avg 0.97 0.97 0.96 - *Table 4: Logistic Regression Results* Precision Recall F1-Score Support Overall 0.85 1.00 0.92 - 1.00 0.49 0.66 - balanced accuracy - - - 0.7452 accuracy - - - 0.8722 macro avg 0.93 0.75 0.79 - weighted avg 0.89 0.87 0.86 - *Table 5: SVM with linear kernel results* Precision Recall F1-Score Support Overall - 1.00 0.54 0.70 - balanced accuracy - - - 0.7710 accuracy - - - 0.8855 macro avg 0.93 0.77 0.82 - weighted avg 0.90 0.89 0.87 - *Table 6: SVM with polynomial kernel results* find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/manoleasa-2022-fii-uaic-semeval-2022-task/