Elamur Research⌕
☰
home / papers / 2022 / TechSSN at SemEval-2022 Task 6: Intended Sarcasm Detection using Transformer Models

TechSSN at SemEval-2022 Task 6: Intended Sarcasm Detection using Transformer Models

Paper · 2022 · Rajalakshmi Sivanaiah, Angel Deborah S, Sakaya Milton R et al.

Summary

The TechSSN team took part in iSarcasmEval (SemEval-2022 Task 6). The task asks whether a short social-media text is sarcastic in the sense its author intended, not how readers perceive it. The team entered three subtasks: Sub-Task A English, Sub-Task A Arabic and Sub-Task B. Sub-Task B is multi-label tagging of sarcasm types such as irony, satire, understatement, overstatement and rhetorical questions.

They used only the organisers' data. Texts were cut into subwords with a BERT-base-uncased tokenizer, limited to 200 tokens, and passed to fine-tuned pre-trained transformers: BERT, ColBERT and RoBERTa.

Scores were modest. ColBERT was best on English, reaching F1 0.2637 and accuracy 0.7407. BERT on Arabic gave F1 0.2292, and RoBERTa on Sub-Task B gave macro-F1 0.0596. The team placed 27th, 29th and 17th.

The paper is mainly a record of what an off-the-shelf fine-tuning baseline achieves on sarcasm labelled by the authors themselves. It also shows how much harder intended sarcasm is than its surface wording suggests.

Key points

  • The English training set has about 3467 texts, roughly 866 of them sarcastic. The Arabic set has about 3102 texts, roughly 746 sarcastic. Test sets contain 1400 (A English), 1400 (A Arabic) and 1616 (B) items.
  • English rows carry binary flags for sarcasm, irony, satire, understatement, overstatement and rhetorical question, plus a non-sarcastic rephrase by the author. Arabic rows add the rephrase and the regional dialect.
  • Preprocessing used BertTokenizer with BERT-base-uncased and a fixed maximum length of 200. The paper describes this tokenizer even for Arabic.
  • Sub-Task A English results:
    • BERT: F1 0.2558, accuracy 0.2936.
    • ColBERT: F1 0.2637, accuracy 0.7407.
  • Sub-Task A Arabic with BERT scored F1 0.2292 and accuracy 0.3707. Sub-Task B with a multi-label RoBERTa scored macro-F1 0.0596. Final ranks were 27th (A English), 29th (A Arabic) and 17th (B).

Interesting ideas

  • I1. The dataset takes sarcasm labels from the authors of the texts rather than from hashtags or outside annotators, which avoids noisy weak supervision and annotator bias.
  • I2. Each sarcastic text is paired with its author's literal rephrase, giving a ready-made pair of what was said and what was meant.
  • I3. Sarcasm types (satire, understatement, overstatement, rhetorical question) are separate binary labels rather than one class, so a text can carry several devices at once.
  • I4. The reported numbers show how accuracy and F1 can disagree sharply on an imbalanced set. About three quarters of the data is non-sarcastic, so ColBERT's 0.74 accuracy sits near the majority share while its F1 stays around 0.26.
  • I5. Using an English uncased tokenizer for Arabic shows how a mismatch at the input stage can quietly cap later results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony detector in prag could be checked against author-intended labels, so that "this was meant ironically" is a target separate from "a reader might take it ironically".
  • I2 (frames): these pairs could test whether prag's implicature step turns the sarcastic surface frame into the same meaning as the parsed rephrase.
  • I3 (frames): prag could emit these devices as independent, explainable flags, each justified by its own cue, instead of a single irony verdict.
  • I4 (learn): every irony or speech-act evaluation should report minority-class F1 next to a majority-class baseline as a negative control.
  • I5 (tokens): if crate en is ever reused on other languages or noisy social text, tokenizer coverage should be checked before any downstream scores are trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sri Sivasubramaniya Nadar College of EngineeringChennai · India2 works · 4 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 worksSemitic languages · 1,484 works

Cite

Cite as Rajalakshmi Sivanaiah, Angel Deborah S, Sakaya Milton R, Mirnalinee T T and Ramdhanush Venkatakrishnan. 2022. TechSSN at SemEval-2022 Task 6: Intended Sarcasm Detection using Transformer Models. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.118

@inproceedings{sivanaiah2022techssn,
  title = {{TechSSN at SemEval-2022 Task 6: Intended Sarcasm Detection using Transformer Models}},
  author = {Rajalakshmi Sivanaiah, Angel Deborah S, Sakaya Milton R, Mirnalinee T T and Ramdhanush Venkatakrishnan},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.118},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.118}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 student research workshop, pages 107–113.
  2. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics.
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  4. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392–398. Citeseer.
  5. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48. ↗
  6. Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective induction of semantic feature vectors. arXiv preprint arXiv:1805.05388. ↗
  7. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. ↗
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  9. Diana G Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Lrec 2014 proceedings. ELRA.
  10. Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 5(4):1093–1113. ↗
  11. Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, and Xiaodan Zhu. 2016. Developing a successful semeval task in sentiment analysis Silviu Vlad Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–22.
  12. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers, pages 213–223.
  13. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  14. Rajalakshmi Sivanaiah, Angel Deborah S, S Milton Rajendram, and Mirnalinee T T. 2018. SSN MLRG1 at SemEval-2018 task 3: Irony detection in English tweets using MultiLayer perceptron. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 633–637, New Orleans, Louisiana. Association for Computational Linguistics. find
  15. Rajalakshmi Sivanaiah, Angel Deborah S, S Milton Rajendram, Mirnalinee TT, Abrit Pal Singh, Aviansh Gupta, and Ayush Nanda. 2021. TECHSSN at SemEval-2021 task 7: Humor and offense detection and classification using ColBERT embeddings. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1185– 1189, Online. Association for Computational Linguistics. find
  16. Rajalakshmi Sivanaiah, Angel Suseelan, Logesh B, Harshini S, Geetika B, Dyaneswaran S, S Milton Rajendram, and Mirnalinee T T. 2019. TECHSSN at SemEval-2019 task 6: Identifying and categorizing offensive language in tweets using deep neural networks. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 753–758, Minneapolis, Minnesota, USA. Association for Computational Linguistics. find
  17. Rajalakshmi Sivanaiah, Angel Suseelan, S Milton Rajendram, and Mirnalinee T.t. 2020. TECHSSN at SemEval-2020 task 12: Offensive language detection using BERT embeddings. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2190–2196, Barcelona (online). International Committee for Computational Linguistics. find
  18. Cynthia Van Hee, Gilles Jacobs, Chris Emmery, Bart Desmet, Els Lefever, Ben Verhoeven, Guy De Pauw, Walter Daelemans, and Véronique Hoste. 2018. Automatic detection of cyberbullying in social media text. PloS one, 13(10):e0203794. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sivanaiah-2022-techssn-semeval-2022-task-6/