Elamur Research⌕
☰
home / papers / 2021 / Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection

Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection

Paper · 2021 · John Culnan, Seongjin Park, Meghavarshini Krishnaswamy et al.

Summary

Multimodal systems for emotion, sarcasm, and personality detection are usually evaluated on human-made transcripts, but a deployed system must rely on automatic speech recognition (ASR). This paper measures how much ASR noise hurts such systems and whether word error rate (WER) predicts that harm. The authors transcribed three English datasets, MELD (emotion), MUStARD (sarcasm), and FirstImpr (OCEAN personality), with two open-source recognizers (PocketSphinx and Kaldi with a Librispeech model) and one commercial service (Google Cloud Speech-to-Text). They compared each against the gold transcripts. They then trained deliberately simple text-only, audio-only, late-fusion multimodal, and multitask models on each transcript version.

Every ASR transcript lowered downstream scores relative to gold, sometimes sharply. Google had by far the lowest WER, but its downstream advantage was small, inconsistent, and not statistically significant. Adding OpenSMILE audio features recovered part of the loss, while naive multitask training did not. The main lesson is that ASR tools should be chosen by downstream task performance rather than by WER alone.

Key points

  • WER for Sphinx / Kaldi / Google:
    • MELD: 114.9 / 104.4 / 82.6
    • MUStARD: 96.8 / 73.0 / 54.3
    • FirstImpr: 97.4 / 83.5 / 63.8
  • Effect on emotion detection: On MELD, the text-only model scores 57.32 with gold transcripts but only 32.68–37.60 with ASR transcripts. The multimodal model with gold reaches 56.28, compared with 37.90–40.94 using ASR text.
  • Sarcasm and personality: On MUStARD, audio alone scores 75.42, the best overall, though not significantly so. On FirstImpr, the multimodal model with Google transcripts (35.98) slightly beats the one with gold (35.05).
  • Rankings diverge from WER: Among ASR tools, Sphinx gives the best MUStARD result despite having the worst WER.
  • Setup:
    • Text model: two-layer LSTM over 300d GloVe (42B) plus 30d trainable embeddings.
    • Audio features: IS10 or IS13 OpenSMILE sets, taken from the middle 50% of each clip.
    • Code and data splits are released (clulab/tomcat-speech).

Interesting ideas

  • I1. Test on realistic input, not gold: Evaluating on the input a real deployment would actually see, rather than on gold input, exposed large hidden drops that clean benchmarks concealed.
  • I2. Rank components by the end task: An upstream component's intrinsic metric (WER) ranked the tools differently from the end task. Candidate components should therefore be compared on the final task as well as on their own score.
  • I3. Lexicon-based error analysis with tokens vs. types: The authors counted emotion-lexicon words (EmoLex) in each transcript, separating tokens from types. This revealed spurious and missing affect words that explain errors better than a global error rate.
  • I4. Keep an independent second channel: A separate signal (audio features) combined by late fusion partly compensated for noisy text. Combining independent evidence makes a system more robust when one input degrades.
  • I5. Exclude overlapping data from multitask training: MUStARD was left out of multitask training because its utterances overlap with MELD, which would leak test material into training.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Run the perceptron parser and gparse on noisy text (ASR-style output, missing punctuation, typos) as well as clean treebanks to measure how much LAS really degrades.
  • I2 (learn): When choosing among witness parsers or taggers, also score each one by its effect on downstream answer accuracy, since a better LAS may not mean better answers.
  • I3 (frames): Count lexicon-flagged words (emotion, irony cues) between clean and degraded text to see which pragmatic signals for irony or sentiment get lost or invented.
  • I4 (frames): Irony and speech-act decisions could merge parse-based evidence with separate cues such as punctuation and lexical markers, so a single parse error does not decide the result.
  • I5 (learn): Before mixing silver corpora or school-test sets for training, check for shared sentences across splits so that evaluations and negative controls stay honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 4Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 worksVision & language · 269 worksSpeech & spoken language · 233 works

Cite

Cite as John Culnan, Seongjin Park, Meghavarshini Krishnaswamy and Rebecca Sharp. 2021. Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection. Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{culnan2021me,
  title = {{Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection}},
  author = {John Culnan, Seongjin Park, Meghavarshini Krishnaswamy and Rebecca Sharp},
  booktitle = {Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2021},
  url = {https://aclanthology.org/2021.wassa-1.26/}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagus Tris Atmaja and Masato Akagi. 2020. Dimensional speech emotion recognition from speech features and word embeddings by using multitask learning. APSIPA Transactions on Signal and Information Processing, 9. ↗
  2. Shahana Bano, Pavuluri Jithendra, Gorsa Lakshmi Ni-harika, and Yalavarthi Sikhi. 2020. Speech to text translation enabling multilingualism. In 2020 IEEE International Conference for Innovation in Technology (INOCON), pages 1–4. IEEE. find
  3. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. Association for Computational Linguistics.
  4. Fabio Celli, Elia Bruni, and Bruno Lepri. 2014. Automatic personality and interaction style recognition from facebook profile pictures. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1101–1104. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association for Computational Linguistics: Human LanguageTech-nologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  6. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the Munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. find
  7. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. arXiv preprint arXiv:1605.02276. ↗
  8. Kallirroi Georgila, Anton Leuski, Volodymyr Yanov, and David Traum. 2020. Evaluation of off-the-shelf speech recognizers across diverse dialogue domains. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6469–6476. find
  9. Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. Cosmic: Commonsense knowledge for emotion identification in conversations. arXiv preprint arXiv:2010.02795. ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  11. Paul Lamere, Philip Kwok, Evandro Gouvea, Bhiksha Raj, Rita Singh, William Walker, Manfred Warmuth, and Peter Wolf. 2003. The cmu sphinx-4 speech recognition system. find
  12. Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Julien Epps, and Bjórn Wolfgang Schuller. 2020. Multi-task semi-supervised adversarial autoencoding for speech emotion recognition. IEEE Transactions on Affective Computing. ↗
  13. Saif Mohammad and Peter Turney. 2010. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, pages 26–34.
  14. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational intelligence, 29(3):436–465. ↗
  15. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. find
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  17. Vı́ctor Ponce-López, Baiyu Chen, Marc Oliu, Ciprian Corneanu, Albert Clapés, Isabelle Guyon, Xavier Baró, Hugo Jair Escalante, and Sergio Escalera. 2016. Chalearn lap 2016: First round challenge on first impressions-dataset and results. In European conference on computer vision, pages 400–418. Springer. find
  18. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527– 536, Florence, Italy. Association for Computational Linguistics.
  19. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. IEEE Signal Processing Society. IEEE Catalog No.: CFP11SRW-USB. find
  20. Esteban Andres Rissola, Seyed Ali Bahrainian, and Fabio Crestani. 2019. Personality recognition in conversations using capsule neural networks. In IEEE/WIC/ACM International Conference on Web Intelligence, pages 180–187. find
  21. Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian Müller, and Shrikanth S Narayanan. 2010. The interspeech 2010 paralinguistic challenge. In Eleventh Annual Conference of the International Speech Communication Association. find
  22. Björn Schuller, Stefan Steidl, Anton Batliner, Alessandro Vinciarelli, Klaus Scherer, Fabien Ringeval, Mohamed Chetouani, Felix Weninger, Florian Eyben, Erik Marchi, et al. 2013. The interspeech 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In Proceedings INTER-SPEECH 2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France. find
  23. Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych. 2018. Multi-task learning for argumentation mining in low-resource settings. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find
  24. Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3–14. ↗
  25. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. ICLR. ↗
  26. Shen Yan, Di Huang, and Mohammad Soleymani. 2020. Mitigating biases in multimodal personality assessment. In Proceedings of the 2020 International Conference on Multimodal Interaction, pages 361–369. find
  27. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018. Multi-attention recurrent network for human communication comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). ↗
  28. Sayyed M Zahiri and Jinho D Choi. 2017. Emotion detection on TV show transcripts with sequence-based convolutional neural networks. In Proceedings ofthe AAAI Workshop on Affective Content Analysis, pages 44–51, New Orleans, LA. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/culnan-2021-me-myself-ire-effects-automatic/