Elamur Research⌕
☰
home / papers / 2022 / UoR-NCL at SemEval-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection

UoR-NCL at SemEval-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection

Summary

The paper presents the UoR-NCL entry to iSarcasmEval (SemEval-2022 Task 6, sub-task A), a binary task of deciding whether English or Arabic tweets are sarcastic. The authors fine-tune BERT with a one-layer feed-forward classifier on the [CLS] representation. Each tweet is joined with a fixed cue segment marking the sarcasm question. Their main change is the training objective. Instead of plain cross-entropy, they use a weighted blend of cross-entropy and negative log-likelihood, tried in two mixes (0.8/0.2 and 0.2/0.8). They also require at least 70% confidence before labelling a tweet sarcastic.

The results are mixed:

  • English validation: the single-loss baseline wins on every metric.
  • Arabic validation: the ensemble wins on most metrics.
  • Test: the ensemble slightly beats the baseline on English F-1 sarcastic. On Arabic it lands marginally below the baseline in the text's figures.

The work matters mainly as a small, reproducible probe of whether changing only the loss helps a hard figurative-language task. In this setup, any gain is small and inconsistent.

Key points

  • Data: the organizers supplied 3467 English and 3120 Arabic training tweets, split 90/10 into train and validation. The test sets totalled 1,400 samples.
  • Setup:
    • Inputs are truncated or padded to 64 tokens.
    • BERT hidden size is 768, with a 120-unit classifier and 2 labels.
    • Training uses Adam for 12 epochs, batch size 32, learning rate 5e-6 and epsilon 1e-9.
  • Loss variants:
    • Ensemble loss A weights cross-entropy and NLL at 80/20.
    • Ensemble loss B reverses this to 20/80.
    • The baseline uses cross-entropy only.
  • English test, F-1 sarcastic: ensemble A scored 79.75% against 79.37% for the baseline.
  • Arabic test, F-1 sarcastic: the text reports 79.12% for the ensemble against 79.74% for the baseline. On Arabic validation, ensemble A scored 83.53% against the baseline's 83.35%.
  • Evaluation: metrics were false positives, recall, F-1 sarcastic, F1-macro and accuracy. ROC curves were used to compare models across thresholds.

Interesting ideas

  • I1. Tweets were labelled sarcastic only above a 70% probability, not the usual 50%; raising the cutoff for a hard, error-prone class trades recall for fewer false alarms. — For Mova (frames): the irony detector in prag could fire only above a strict confidence bar, so ironic re-readings of a clause are applied only when evidence is strong.
  • I2. The authors report validation and test results separately and openly show that the loss tweak helps on one split or language but not another, a useful reminder to check that small gains are stable. — For Mova (learn): any new training objective or repair rule should be accepted only if it improves on several held-out splits, alongside negative controls.
  • I3. A fixed cue segment ("sarcastic sentence") is placed after [SEP], turning plain classification into a paired input that names the question being asked. — For Mova (frames): when a transformer scores speech acts or irony, a short task-naming second segment could let one encoder serve several pragmatic questions.
  • I4. The objective is a weighted mix of two losses whose proportions are treated as a tunable knob, with more mixes left for future search. — For Mova (tree): gparse training could blend arc and label losses (or gold and silver losses) with explicit weights tuned on dev data, instead of a fixed sum.
  • I5. ROC curves are drawn to compare models over all thresholds, not just at one operating point. — For Mova (answer): reader confidence in answers could be judged with threshold-free curves, so the chosen abstain cutoff is a visible, explainable choice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Sentiment & emotion · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.121

@inproceedings{oseibrefo2022uor,
  title = {{UoR-NCL at SemEval-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection}},
  author = {Osei-Brefo, Emmanuel and Liang, Huizhi},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.121},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.121}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find our card
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoRR, abs/1607.00976. ↗ our card
  3. Adithya Avvaru, Sanath Vobilisetty, and Radhika Mamidi. 2020. Detecting Sarcasm in Conversation Context Using Transformer-Based Models. In Proceedings of the Second Workshop on Figurative Language Processing, pages 98–103, Online. Association for Computational Linguistics. find our card
  4. Imran Sarwar Bajwa and Muhammad Abbas Choudhary. 2006. A rule based system for speech language context understanding. Journal of Donghua University (English Edition) Vol, 23(06). find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. ↗
  6. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 186–196, Saarbrücken, Germany. Association for Computational Linguistics. find our card
  7. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? find our card
  8. Akshay Khatri and Pranav P. 2020. Sarcasm detection in tweets with BERT and GloVe embeddings. In Proceedings of the Second Workshop on Figurative Language Processing, pages 56–60, Online. Association for Computational Linguistics. find
  9. Bing Liu. 2010. Sentiment analysis and subjectivity. In Handbook of Natural Language Processing, Second Edition. Taylor and Francis Group, Boca. find
  10. Paul K Mandal and Rakeshkumar Mahto. 2019. Deep cnn-lstm with word embeddings for news headline sarcasm detection. In 16th International Conference on Information Technology-New Generations (ITNG 2019), pages 495–498. Springer. find
  11. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  12. Bleau Moores and Vijay Mago. 2022. A survey on automated sarcasm detection on twitter. find
  13. Emmanuel Osei-Brefo, Thanet Markchom, and Huizhi Liang. 2021. UOR at SemEval-2021 task 12: On crowd annotations; learning with disagreements to optimise crowd truth. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1303–1309, Online. Association for Computational Linguistics. find
  14. Krishna Parmar, Nivid Limbasiya, and Maulik V. Dhamecha. 2018. Feature based composite approach for sarcasm detection using mapreduce. 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), pages 587–591. find
  15. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm - a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM. find
  16. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460, Osaka, Japan. find our card
  17. The COLING 2016 Organizing Committee. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/brefo-2022-uor-ncl-semeval-2022-task/