Elamur Research⌕
☰
home / papers / 2022 / Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification

Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification

Paper · 2022 · Tulika Saha, Aditya Prakash Patra, Sriparna Saha et al.

Summary

The paper asks whether knowing a speaker's sentiment and emotion helps identify the dialogue act (DA) of an utterance. Examples are sarcastic "agreement" that is really disagreement, or an apology that is easily mistaken for an opinion. The authors added utterance-level sentiment labels (positive/negative/neutral) to the multi-modal EMOTyDA corpus, which covers text, audio and video and already had DA and emotion tags. They then trained one network for three tasks at once: DAC, sentiment analysis (SA) and emotion recognition (ER). Each modality is encoded by its own Bi-LSTM. A Dual Attention Mechanism (DAM) combines attention within each modality with interactive attention across modalities, and losses are attached at several depths of the network. DA classes are heavily skewed, so the authors add Meta-Net (MN) learning, in which a small auxiliary network learns per-sample loss weights from the data. They also add a 2-step Deferred Optimisation Schedule (DOS), which switches on re-weighting only after an initial training phase, using a lower learning rate. The full system beats earlier fusion models and the prior emotion-aware DAC state of the art. This supports treating affect as a useful signal for pragmatics.

Key points

  • EMOTyDA is built from MELD and IEMOCAP conversations: 1341 dialogues, 19,365 utterances, about 22 hours, 12 DA tags and 10 emotion tags.
  • Sentiment labels from MELD were reused as released. IEMOCAP utterances were labelled by three annotators who watched the video with its context, reaching 78% inter-annotator agreement.
  • Input features:
    • text: GloVe, 300-d;
    • audio: openSMILE with the INTERSPEECH 2009 emotion config, 384-d;
    • video: ImageNet-pretrained ResNet-152, 4096-d.
  • Text+video was the strongest modality pair. It edged out using all three modalities (59.50 vs 58.73 accuracy).
  • Results with the best configuration (text+video):
    • DAC+SA+ER: 59.50% accuracy, 54.86 F1;
    • adding MN+DOS: 61.72% accuracy, 57.01 F1;
    • previous best emotion-aided DAC model: 56.62% accuracy, 51.70 F1.

Interesting ideas

  • I1. Delay class re-weighting: first train plainly so the model forms general representations, then fine-tune with a re-weighted loss at a smaller learning rate, avoiding early distortion by rare classes.
  • I2. Rather than hand-picking a loss-to-weight mapping for imbalanced data, a tiny meta-network learns how much each training sample should count.
  • I3. Predicting sentiment and emotion as auxiliary signals helps resolve speech acts whose literal wording points one way and whose intent points another, notably sarcastic agreement.
  • I4. The error analysis notes that neutral sentiment or emotion contributes nothing, and that wrongly predicted affect drags DA predictions down with it, so auxiliary cues can harm as well as help.
  • I5. Supervising all tasks at several layers of the network, not only at the output, pushes intermediate representations to be shared across tasks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When training gparse or the tagger on skewed silver data (rare relations, rare speech acts), a second re-weighted phase could lift rare labels without destabilising the base model.
  • I2 (learn): Could weight silver sentences by learned trust instead of fixed witness-agreement thresholds, with negative controls checking that the learned weights are meaningful.
  • I3 (frames): The speech-act and irony modules in prag could consult an explicit, explainable polarity/affect feature of the utterance as evidence for flipping "agree" to "disagree".
  • I4 (frames): Irony and implicature rules should fire on affect only when the affect signal is confident and non-neutral, and record that condition in the explanation trace.
  • I5 (tree): For a multi-head gparse (tags, heads, relations), attaching auxiliary losses at intermediate layers is a cheap experiment toward sturdier shared encodings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology BombayMumbai · India22 works · 2,018 citations🎓Indian Institute of Technology PatnaPatna · India10 works · 262 citations🎓University of LiverpoolLiverpool · United Kingdom7 works · 539 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 608 worksVision & language · 269 worksDialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Tulika Saha, Aditya Prakash Patra, Sriparna Saha and Pushpak Bhattacharyya. 2022. Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification. Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.aacl-main.71

@inproceedings{saha2022meta,
  title = {{Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification}},
  author = {Tulika Saha, Aditya Prakash Patra, Sriparna Saha and Pushpak Bhattacharyya},
  booktitle = {Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-main.71},
  url = {https://doi.org/10.18653/v1/2022.aacl-main.71}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa Feldman. Barrett, Michael Lewis, and Jeannette M. Haviland-Jones. 1993. Handbook of emotions. The Guilford Press. find
  2. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. In this paper, we study the role of sentiment and emotion while modelling the task of DAC. For this, we propose a Dual Attention Mechanism based multi-modal, multi-tasking framework for jointly Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, optimizing DAC, SA and ER tasks. ↗
  3. The DAM and Tengyu Ma. 2019. Learning imbalanced datasets module employs intra-modal and interactive inter-with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems 32: modal attentions with multiple loss optimization at Annual Conference on Neural Information Process-various hierarchies in order to fuse multiple modali-ing Systems 2019, NeurIPS 2019, December 8-14, ties efficiently and learn generalized features across 2019, Vancouver, BC, Canada, pages 1565–1576. all the tasks. Additionally, to counter the class-Santiago Castro, Devamanyu Hazarika, Verónica Pérez-imbalance issue in dialogues, we introduce a 2-step Rosas, Roger Zimmermann, Rada Mihalcea, and Sou-DOS that involves MN learning and deferred re-janya Poria. 2019. Towards multimodal sarcasm weighting where the former is an adaptive sam-detection (an _obviously_ perfect paper). In Pro-ple weighting strategy to automatically learn an ceedings of the 57th Conference of the Association explicit weighting function from data and the lat-for Computational Linguistics, ACL 2019, Florence, 986 find
  4. Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4619–4629. find
  5. Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa Le. 2018a. Multi-task dialog act and sentiment recognition on mastodon. arXiv preprint arXiv:1807.05013. ↗
  6. Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa T. Le. 2018b. Multi-task dialog act and sentiment recognition on mastodon. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 745– 754. ↗
  7. Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. 2016. Learning deep representation for imbalanced classification. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5375–5384. IEEE Computer Society. find
  8. Peter Lajos Ihasz and Victor Kryssanov. 2018. Emotions and intentions mediated with dialogue acts. In 2018 5th International Conference on Business and Industrial Research (ICBIR), pages 125–130. IEEE. find
  9. Anurag Illendula and Amit Sheth. 2019. Multimodal emotion classification. In Companion Proceedings of The 2019 World Wide Web Conference, pages 439– 449. find
  10. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. 2019. Class-balanced loss based on effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9268–9277. Computer Vision Foundation / IEEE. find
  11. Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2012–2021.
  12. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. find
  13. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. In Thirty-Second AAAI Conference on Artificial Intelligence, pages 3440–3447. ↗
  14. Hai Ha Do, PWC Prasad, Angelika Maag, and Abeer Al-sadoon. 2019. Deep learning for aspect-based sentiment analysis: a comparative review. Expert Systems with Applications, 118:272–299. ↗
  15. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. ACM. find
  16. Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. 2018. Bilevel programming for hyperparameter optimization and meta-learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1563–1572. PMLR. ↗
  17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. ↗
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  19. M Shamim Hossain and Ghulam Muhammad. 2019. Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion, 49:69–78. ↗
  20. M. Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pages 1189–1197. Curran Associates, Inc. find
  21. Jingye Li, Hao Fei, and Donghong Ji. 2020. Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions. In Proceedings of the 28th International Conference on Computational Linguistics, pages 616– 626. ↗
  22. Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2999–3007. IEEE Computer Society. ↗
  23. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in DNN framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170–2178, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  24. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818–6825. ↗
  25. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. 2017. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5947–5956. find
  26. Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. 2018. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. ↗
  27. Pinelopi Papalampidi, Elias Iosif, and Alexandros Potamianos. 2017. Dialogue act semantic representation and classification using recurrent neural networks. SEMDIAL 2017 SaarDial, page 104. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  29. Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2539–2544. ↗
  30. Soujanya Poria, Erik Cambria, Newton Howard, Guang-Bin Huang, and Amir Hussain. 2016. Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing, 174:50–59. ↗
  31. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 527–536.
  32. Libo Qin, Wanxiang Che, Yangming Li, Mingheng Ni, and Ting Liu. 2020. Dcr-net: A deep co-interactive relation network for joint dialog act recognition and sentiment classification. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8665–8672.
  33. Tulika Saha and Sophia Ananiadou. 2022. Emotion-aware and intent-controlled empathetic response generation using hierarchical transformer network. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  34. Tulika Saha, Dhawal Gupta, Sriparna Saha, and Pushpak Bhattacharyya. 2020a. Emotion aided dialogue act classification for task-independent conversations in a multi-modal framework. Cognitive Computation, pages 1–13. ↗
  35. Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020b. Towards emotion-aided multimodal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361–4372. find
  36. Tulika Saha, Aditya Prakash Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020c. A transformer based approach for identification of tweet acts. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  37. Tulika Saha, Srivatsa Ramesh Jayashree, Sriparna Saha, and Pushpak Bhattacharyya. 2020d. Bert-caps: A transformer-based capsule network for tweet act classification. IEEE Transactions on Computational Social Systems, 7(5):1168–1179. ↗
  38. Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2019. Tweet act classification : A deep learning based classifier for recognizing speech acts in twitter. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  39. Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2020e. Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning. PloS one, 15(7):e0235367. ↗
  40. Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2020f. Towards sentiment-aware multi-modal dialogue policy learning. Cognitive Computation, pages 1–15. ↗
  41. Tulika Saha, Apoorva Upadhyaya, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Towards sentiment and emotion aided multi-modal speech act classification in Twitter. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5727–5737, Online. Association for Computational Linguistics.
  42. Tulika Saha, Apoorva Upadhyaya, Sriparna Saha, and Pushpak Bhattacharyya. 2022. A multitask multimodal ensemble model for sentiment- and emotion-aided tweet act classification. IEEE Transactions on Computational Social Systems, 9(2):508–517. ↗
  43. Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weight-net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1917–1928. find
  44. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗
  45. Yanmin Sun, Mohamed S. Kamel, Andrew K. C. Wong, and Yang Wang. 2007. Cost-sensitive boosting for classification of imbalanced data. Pattern Recognit., 40(12):3358–3378. ↗
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–6008.
  47. Yixin Wang, Alp Kucukelbir, and David M. Blei. 2017. Robust probabilistic modeling with bayesian data reweighting. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3646–3655. PMLR. ↗
  48. Bernard L Welch. 1947. The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika, 34(1/2):28–35. find
  49. Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Jian-Huang Lai, and Tie-Yan Liu. 2018. Learning to teach with dynamic loss functions. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6467–6478. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saha-2022-meta-learning-based-deferred-optimisation/