Elamur Research⌕
☰
home / papers / 2020 / A Novel Hierarchical BERT Architecture for Sarcasm Detection

A Novel Hierarchical BERT Architecture for Sarcasm Detection

Summary

This workshop paper addresses context-aware sarcasm detection: deciding whether a reply in an online conversation is sarcastic, given the preceding conversation. The authors used the Twitter and Reddit datasets from the FigLang-2020 shared task. Their model handles the context hierarchically:

  • One BERT encoder represents each context utterance, and a second BERT encoder plus a BiLSTM represents the response.
  • A 2D convolution compresses the very large stack of context representations.
  • A BiLSTM models how the conversation unfolds turn by turn.
  • A Kim-style multi-kernel convolution with max-pooling relates the context to the response, and a fully connected sigmoid layer gives the final decision.

The model was compared with a Hierarchical Attention Network and a Memory Network baseline (KG-Mem). It scored higher on both datasets, which the authors take as evidence that structured use of conversation history beats sequential processing. Error analysis using attention visualisation and t-SNE suggests that most mistakes are borderline cases, such as positive wording used negatively. Many test samples fall in that ambiguous region, which may explain why test scores are lower than validation scores.

Key points

  • Data: FigLang-2020 shared task data. Twitter has 5000 training instances and Reddit has 4400, with 1800 test instances each. The training data was split 80:20 (stratified) into train and dev sets.
  • Architecture: separate BERT encoders for context and response, with d_bert = 768 and maximum sentence length 100. The rest of the pipeline is conv2d context summarization with a (2,2) kernel, a BiLSTM context encoder, and CNN kernels of size (2,2), (2,3) and (2,5).
  • Test F1, Twitter: 0.74 for the proposed model, compared with 0.68 for HAN and 0.70 for KG-Mem.
  • Test F1, Reddit: 0.639 for the proposed model, compared with 0.60 for HAN and 0.605 for KG-Mem.
  • Training and future work: the model was trained with cross-entropy loss and the Adam optimizer, and tuned on validation F1. The authors plan to find more efficient ways to compress BERT context representations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.14

@inproceedings{srivastava2020novel,
  title = {{A Novel Hierarchical BERT Architecture for Sarcasm Detection}},
  author = {Srivastava, Himani and Varshney, Vaibhav and Kumari, Surabhi and Srivastava, Saurabh},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.14},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.14}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579–2605. find
  2. Silvio Amir, Byron Wallace, Hao Lyu, Paula Carvalho, and Mário Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. pages 167–177. find our card
  3. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. ArXiv, 1409. ↗ our card
  5. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014. Italian irony detection in twitter: a first approach. find
  6. Jakub Nowak, Ahmet Taspinar, and Rafal Scherer. 2017. Lstm recurrent neural networks for short text and sentiment classification. pages 553–562. find
  7. S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 1373–1380. find
  8. Rolandos Potamias, Georgios Siolas, and Andreas Stafylopatis. 2019. A transformer-based approach to irony and sarcasm detection. ↗ our card
  9. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. pages 2506–2515. find our card
  10. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation, 47. ↗
  11. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078. ↗
  12. Ellen Riloff, A. Qadir, P. Surve, L. Silva, N. Gilbert, and R. Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. Proceedings of EMNLP, pages 704–714. find our card
  13. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  15. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448. ↗
  16. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  17. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm - a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9:1735– 80. ↗
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  20. Aditya Joshi, Kevin Patel, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? ↗ our card
  21. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. pages 765–770. find
  22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. find our card
  23. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489. find our card
  24. Christine Liebrecht, Florian Kunneman, and Antal van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37, Atlanta, Georgia. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/srivastava-2020-novel-hierarchical-bert-architecture-sarcasm/