Elamur Research⌕
☰
home / papers / 2020 / EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators

EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators

Paper · 2020 · Chandrakant Bothe, Cornelius Weber, Sven Magg et al.

Summary

Conversational emotion corpora label each utterance with an emotion but say nothing about its dialogue act, the intention or function an utterance performs, such as asking, agreeing, or thanking. Dialogue systems and discourse analysis need both layers. The authors fill this gap automatically. They trained five recurrent neural classifiers with attention on the DAMSL-tagged Switchboard Dialogue Act (SwDA) corpus, using pre-trained ELMo embeddings. Some classifiers look only at the current utterance, and others also see the two preceding turns. An ensemble merges their outputs, first by majority vote and then by ranking model confidence. On the SwDA test set it scored slightly above the best single model. The ensemble then labelled two multimodal emotion corpora, IEMOCAP and MELD. The authors checked label reliability with inter-annotator agreement measures and released the result as the Emotional Dialogue Acts (EDA) corpus. Cross-tabulating the two label layers revealed systematic pairings between intentions and feelings. This makes EDA a resource for building dialogue agents that respond to both what the user means and how they feel.

Key points

  • The annotator pool has five members trained on SwDA:
    • Utt-level-1 (RNN + attention), 75.1% accuracy.
    • Context-1 (hierarchical RNN over the current and two prior utterances), 77.55% accuracy, the strongest single model.
    • Utt-level-2 (mean of embeddings + feed-forward layer), 72.6% accuracy.
    • Context-2, 75.9% accuracy.
    • Context-3, 74.9% accuracy.
  • The ensemble reached 0.778 accuracy on the SwDA test set. Spearman correlation with gold labels was also reported for every model (ensemble 0.822).
  • Words are represented by 1024-dimensional pre-trained ELMo vectors. The public Discourse-Wizard demo uses cheaper, model-trained embeddings instead.
  • The target corpora are IEMOCAP (Ekman categories plus Frustration and Excited, and valence/arousal/dominance) and MELD (7 emotion categories plus sentiment). Their utterances are not split into DAMSL segments.
  • Observed pairings between dialogue acts and emotions:
    • Accept/Agree with Joy
    • Reject with Anger
    • Acknowledgement with Surprise
    • Thanking with Joy
    • Apology with Sadness

Interesting ideas

  • I1. The pool deliberately mixes architectures (attention RNN, embedding averaging, context vs. no context), so model errors are less correlated and a majority vote becomes more meaningful. Confidence ranking only breaks ties when no majority exists.
  • I2. Before projecting labels onto an unlabelled corpus, the authors scored each annotator and the ensemble against a gold test set from the source domain. They reported a rank correlation alongside accuracy.
  • I3. Dialogue-act recognition improves with a small, fixed window of the two preceding utterances, without encoding the whole dialogue.
  • I4. Overlaying an automatically added label layer on an existing human-labelled one turns two separate corpora into a resource for studying how the layers relate. Examples are apologies co-occurring with sadness and rejections with anger.
  • I5. The emotion corpora keep multi-function utterances unsegmented, so a statement followed by a question gets a single label. The paper explicitly flags this as a granularity mismatch with DAMSL.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers for silver data could be chosen for architectural diversity, with a two-step merge rule: agreement first, highest calibrated confidence as a fallback.
  • I2 (learn): Every silver-labelling run could be preceded by a gold-set check per witness and for the merged output, with the numbers logged next to the silver data.
  • I3 (frames): The speech-act classifier in prag could use features from the previous two turns or sentences, which keeps its decisions local and explainable.
  • I4 (frames): Speech-act labels and emotion or attitude cues in tales could be cross-tabulated to derive explainable priors, for example treating an apology as evidence of a sad or regretful speaker state.
  • I5 (frames): Speech acts should be assigned per clause or segment from the UD tree rather than per sentence, so mixed utterances yield several acts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany39 works · 4,669 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 608 worksDialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Chandrakant Bothe, Cornelius Weber, Sven Magg and Stefan Wermter. 2020. EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1912.00819

@inproceedings{bothe2020eda,
  title = {{EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators}},
  author = {Chandrakant Bothe, Cornelius Weber, Sven Magg and Stefan Wermter},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  eprint = {1912.00819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.00819}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, J. and Core, M. (1997). Draft of DAMSL: Dialogue Act Markup in Several Layers. Carnegie Mellon University. find
  2. Austin, J. L. (1962). How to Do Things with Words. Oxford University Press. find
  3. Berg, M. M. (2015). NADIA: A Simplified Approach Towards the Development of Natural Dialogue Systems. In International Conference on Applications of Natural Language to Information Systems, pages 144–150. Springer. find
  4. Bothe, C., Magg, S., Weber, C., and Wermter, S. (2018a). Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks. In Proceedings of the International Conference INTERSPEECH 2018, pages 996–1000. International Speech Communication Association (ISCA). find
  5. Bothe, C., Magg, S., Weber, C., and Wermter, S. (2018b). Discourse-Wizard: Discovering Deep Discourse Structure in your Conversation with RNNs. preprint arXiv:1806.11420. ↗
  6. Bothe, C., Weber, C., Magg, S., and Wermter, S. (2018c). A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, pages 1952–1957. European Language Resources Association (ERLA).
  7. Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359. ↗
  8. Chen, S.-Y., Hsu, C.-C., Kuo, C.-C., Huang, T.-H., and Ku, L.-W. (2018). EmotionLines: An Emotion Corpus of Multi-Party Conversations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, pages 1597–1601. ↗
  9. Cowie, R. and Cornelius, R. R. (2003). Describing the Emotional States That Are Expressed in Speech. Speech Communication, 40(1-2):5–32. ↗
  10. Ekman, P., Friesen, W. V., O’sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987). Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion. Journal of Personality and Social Psychology, 53(4):712–717. ↗
  11. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378. ↗
  12. Godfrey, J. J., Holliman, E. C., and McDaniel, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 517–520. find
  13. Ihasz, P. L. and Kryssanov, V. (2018). Emotions and Intentions Mediated with Dialogue Acts. In Proceedings of the 5th International Conference on Business and Industrial Research (ICBIR), pages 125–130. IEEE. find
  14. Jurafsky, D., Shriberg, E., and Biasca, D. (1997). Switchboard Dialog Act Corpus. Technical report, International Computer Science Inst. Berkeley CA. find
  15. Jurafsky, D. (1997). Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual, draft 13. Technical Report 97-01, University of Colorado Institute of Cognitive Science, pages 225–233. find
  16. Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. v. d., Graves, A., and Kavukcuoglu, K. (2016). Neural Machine Translation in Linear Time. arXiv:1610.10099. ↗
  17. Krippendorff, K. (1970). Estimating the Reliability, Systematic Error and Random Error of Interval Data. Educational and Psychological Measurement, 30(1):61–70. ↗
  18. Lakomkin, E., Zamani, M. A., Weber, C., Magg, S., and Wermter, S. (2019). Incorporating End-to-End Speech Recognition Models for Sentiment Analysis. In International Conference on Robotics and Automation (ICRA), pages 7976–7982. IEEE. find
  19. McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276–282. ↗
  20. McKeown, G., Valstar, M., Cowie, R., Pantic, M., and Schroder, M. (2012). The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent. IEEE Transactions on Affective Computing, 3(1):5–17. ↗
  21. Osgood, C. E., May, W. H., Miron, M. S., and Miron, M. S. (1975). Cross-cultural Universals of Affective Meaning, volume 1. University of Illinois Press. find
  22. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep Contextualized Word Representations. North American Chapter of the Association for Computational Linguistics (NAACL 2018), pages 2227–2237.
  23. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2019). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527–536. Association for Computational Linguistics.
  24. Powers, D. M. (2012). The Problem with Kappa. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 345–355. Association for Computational Linguistics. ↗
  25. Russell, J. A. and Mehrabian, A. (1977). Evidence for a Three-Factor Theory of Emotions. Journal of Research in Personality, 11(3):273–294. ↗
  26. Searle, J. R. (1979). Expression and Meaning: Studies in the Theory of Speech Acts. Cambridge University Press. find
  27. Skantze, G. (2007). Error Handling in Spoken Dialogue Systems-Managing Uncertainty, Grounding and Miscommunication: Chapter 2, Spoken Dialogue Systems. KTH Computer Science and Communication. find
  28. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C., and Meteer, M. (2000). Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339–373. ↗
  29. Ultes, S., Rojas Barahona, L. M., Su, P.-H., Vandyke, D., Kim, D., Casanueva, I., Budzianowski, P., Mrkšić, N., Wen, T.-H., Gasic, M., and Young, S. (2017). PyDial: A Multi-domain Statistical Dialogue System Toolkit. In Proceedings of the ACL 2017, System Demonstrations, pages 73–78. Association for Computational Linguistics. find
  30. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arXiv:1804.07461. ↗
  31. Wermter, S. and Löchel, M. (1996). Learning dialog act processing. In Proceedings of the 16th Conference on Computational Linguistics, volume 2, pages 740–745. Association for Computational Linguistics. find
  32. Wood, I., McCrae, J. P., Andryushechkin, V., and Buitelaar, P. (2018). A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, pages 1197–1202. find
  33. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in neural information processing systems, pages 5754–5764. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bothe-2020-eda-enriching-emotional-dialogue-acts/