Elamur Research⌕
☰
home / papers / 2020 / Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media

Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media

Summary

Sarcasm often hides negative intent behind wording that looks positive. On social media it is further obscured by typos, slang and emojis, and a single reply judged in isolation can be ambiguous even to people. Working within the FigLang 2020 sarcasm shared task, the authors test whether feeding the surrounding conversation into a transformer improves detection. They compare two set-ups:

  • a target-oriented classifier that encodes only the reply being judged;
  • a context-aware variant that joins the reply, a separator token and the concatenated other thread turns into one sequence, so self-attention can relate them.

In both, the leading special-token embedding goes to a linear layer for a binary decision. BERT-Large, RoBERTa-Large and ALBERT-xxLarge are fine-tuned on pooled Twitter and Reddit data. RoBERTa-Large performed best. Adding context raised test F1 by 3.1 points on Twitter and 7.0 on Reddit, and the submitted system scored 79.0 and 75.0, ranking second on both leaderboards at submission time. The results show that thread context matters, especially when test conversations are much longer than training ones.

Key points

  • Data: Twitter threads were labelled via #sarcasm/#sarcastic hashtags. The Reddit threads are a subset of the Self-Annotated Reddit Corpus (Khodak et al., 2017). Labels are SARCASM / NOT_SARCASM.
  • Training set sizes: 5,000 Twitter and 4,400 Reddit conversations, with 1,800 test conversations each. The authors dropped 4.82% duplicate tweets, leaving 4,759, and held out 10% of each set as a development set.
  • Development set: On the combined Twitter + Reddit development set, RoBERTa-Large reached 80.2 F1 without context and 81.3 F1 with context, a gain of 1.1.
  • Test results: RoBERTa-Large F1 rose from 75.2 to 78.3 on Twitter and from 67.4 to 74.4 on Reddit. The organizers' LSTM-attention baselines were 67.0 (Twitter) and 60.0 (Reddit), and the task had 36 participating teams.
  • Setup and code: Learning rate 3e-5, 30 epochs, three seeds (21/42/63), and maximum sequence length 128 (target-only) or 256 (with context). Code is available at github.com/emorynlp/figlang-shared-task-2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.38 · arXiv 2005.11424

@inproceedings{dong2020transformer,
  title = {{Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media}},
  author = {Dong, Xiangjue and Li, Changmao and Choi, Jinho D.},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.38},
  eprint = {2005.11424},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.38}
}

References

15 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman and Noah Smith. 2015. Contextualized Sarcasm Detection on Twitter. In International AAAI Conference on Web and Social Media, pages 574–577. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186. find
  3. Aniruddha Ghosh and Tony Veale. 2017. Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482–491, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  4. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The Role of Conversation Context for Sarcasm Detection in Online Interactions. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 186– 196. find our card
  5. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2018. Sarcasm Analysis using Conversation Context. Comput. Linguist., 44(4):755–792. ↗ our card
  6. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003–1012, Lisbon, Portugal. Association for Computational Linguistics. find our card
  7. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Computing Surveys, 50(5):1–22. ↗ our card
  8. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing Context Incongruity for Sarcasm Detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. find our card
  9. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are Word Embedding-based Features Useful for Sarcasm Detection? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1006–1011, Austin, Texas. Association for Computational Linguistics. ↗ our card
  10. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A Large Self-Annotated Corpus for Sarcasm. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), abs/1704.05579. ↗ our card
  11. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv, 11942(1909). find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A Robustly Optimized BERT Pretraining Approach. In Proceedings of the International Conference on Learning Representations. find
  13. Shereen Oraby, Vrindavan Harrison, Amita Misra, Ellen Riloff, and Marilyn Walker. 2017. Are you serious?: Rhetorical Questions and Sarcasm in Social Media Dialog. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 310–319, Saarbrücken, Germany. Association for Computational Linguistics. find our card
  14. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  15. Zelin Wang, Zhijian Wu, Ruimin Wang, and Yafeng Ren. 2015. Twitter Sarcasm Detection Exploiting a Context-Based Model. In WISE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2020-transformer-based-context-aware-sarcasm/