Elamur Research⌕
☰
home / papers / 2020 / Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis

Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis

Summary

Sarcasm often hides a negative attitude behind positive wording, so the authors examine whether a speaker's sentiment and emotion can help a model detect sarcasm in multi-modal dialogue. They first extended the MUStARD dataset, a collection of TV-show clips labelled for sarcasm, by manually adding sentiment and emotion labels. Each label was given in two versions: explicit, read from the utterance alone, and implicit, inferred with help from the preceding context. They then built a deep multi-task network that takes text, acoustic and visual input. Sarcasm is the main task, and the four sentiment/emotion labels act as auxiliary tasks. Modalities are fused with two new attention mechanisms. Ie-Attention links different segments across modalities, while Ia-Attention links the same segment across modalities. On the tri-modal setting, the joint model modestly but significantly outperforms the earlier SVM baseline, and the gain is larger when test speakers are unseen. This supports the idea that affective signals help sarcasm recognition. The work also provides a reusable annotated resource.

Key points

  • Dataset: MUStARD contains 690 class-balanced utterances with context, totalling roughly 3.68 hours drawn from four sources: The Golden Girls, Sarcasmaholics Anonymous, Friends and The Big Bang Theory. It was re-annotated as follows:
    • sentiment uses 3 classes (positive, negative, neutral);
    • emotion uses 9 multi-label classes, including anger, disgust, happy and frustrated;
    • every label has an implicit and an explicit variant.
  • Annotation: Three graduate annotators labelled the data, with final labels chosen by majority vote. Agreement reached a Fleiss' kappa of 0.81. The extended data, guidelines and code are released by IIT Patna.
  • Method: The network shares the fused Ie-Attention and Ia-Attention representations across all five outputs. It uses fastText embeddings for text, plus optional context and speaker information.
  • Results (T+A+V, speaker-dependent): Precision is 73.4, recall 72.8 and F1 72.6, compared with 71.9, 71.4 and 71.5 for the Castro et al. (2019) baseline. In the speaker-independent setting, F1 rises from 62.8 to 65.9, with p < 0.05.
  • Limitations: Multi-tasking hurt performance in the bi-modal T+A setting, possibly because visual affect cues were missing. The authors also note that the dataset is small for complex models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

33 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 30 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.401

@inproceedings{chauhan2020sentiment,
  title = {{Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis}},
  author = {Chauhan, Dushyant Singh and R, Dhanush S and Ekbal, Asif and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.401},
  url = {https://doi.org/10.18653/v1/2020.acl-main.401}
}

References

12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multi-task learning for multi-modal emotion recognition and sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 370–379, Minneapolis, Minnesota. Association for Computational Linguistics. the 57th Annual Meeting of the Association for Computational Linguistics, pages 481–492. ↗
  2. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. arXiv preprint arXiv:1610.08815. ↗ our card
  3. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. ↗
  4. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). arXiv preprint arXiv:1906.01815. ↗ our card
  5. Dushyant Singh Chauhan, Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Context-aware interactive attention for multi-modal sentiment and emotion analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5651–5661, Hong Kong, China. Association for Computational Linguistics. find
  6. KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259. ↗ our card
  7. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many rater. Psychological Bulletin, 76:378–382. find
  8. Deepanway Ghosal, Md Shad Akhtar, Dushyant Singh Chauhan, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Contextual inter-modal attention for multi-modal sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3454–3466, Brussels, Belgium. Association for Computational Linguistics. find
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. ↗
  10. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5):73:1–73:22. ↗ our card
  11. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗ our card
  12. Sijie Mai, Haifeng Hu, and Songlong Xing. 2019. Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing. In Proceedings of find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/chauhan-2020-sentiment-emotion-help-sarcasm-multi/