Elamur Research⌕
☰
home / papers / 2023 / Cross-Corpus Multilingual Speech Emotion Recognition: Amharic vs. Other Languages

Cross-Corpus Multilingual Speech Emotion Recognition: Amharic vs. Other Languages

Summary

The paper looks at speech emotion recognition (SER) for a language with very little labelled data. It asks whether a classifier for Amharic can be trained on emotional speech from other languages. The authors use their own public Amharic Speech Emotion Dataset (ASED) together with RAVDESS (English), EMO-DB (German) and URDU (Urdu). Each corpus has its own emotion labels, so all of them are reduced to two valence classes, positive and negative. This makes the four languages directly comparable and lets them be mixed freely. Three CNN classifiers are tested: AlexNet, ResNet50 and VGGE, a VGG variant the authors propose. There are three experiments: monolingual training, transfer between Amharic and one other language in both directions, and training on several non-Amharic languages before testing on Amharic. Pooling two or three source languages gave several percent higher accuracy than the best single-source transfer, and English or German were the most helpful single sources. This suggests that combining foreign corpora is a workable way to build SER for low-resource languages.

Key points

  • ASED, from the authors' earlier work, contains 2,474 recordings from 65 speakers (25 male, 40 female), each labelled with one of five emotion categories: anger, happiness, sadness, fear, or a neutral state. It is available on GitHub (Ethio2021/ASED_V1).
  • The comparison corpora are RAVDESS (1,440 recordings, 24 participants, 8 emotions), EMO-DB (535 recordings, 10 participants, 7 emotions) and URDU (400 recordings, 38 participants, 4 emotions).
  • Neutral and happy-type labels (plus calm and surprise in RAVDESS) count as positive valence. Fear, sadness, anger, disgust and boredom count as negative valence.
  • In Experiment 1, results averaged over AlexNet, VGGE and ResNet50 were about equal for ASED and RAVDESS. German came out harder and Urdu easier.
  • Cross-lingual transfer was tested in both directions for Amharic↔German, Amharic↔English and Amharic↔Urdu. Multi-source training on non-Amharic data beat every single-source result on Amharic.

Interesting ideas

  • I1. Mapping incompatible label inventories onto one small shared scheme makes corpora with different annotation traditions comparable and poolable, at the cost of finer distinctions. — For Mova (tree): The same move could guide UD dialect conversion: pick a coarse common core of relations that every treebank can be mapped to faithfully, then merge treebanks on that core before refining.
  • I2. Averaging over several unrelated architectures shows how hard a dataset or language is, rather than how well one model happens to fit it. — For Mova (learn): Evaluations could report scores averaged over the perceptron parser and gparse, separating "this genre or treebank is hard" from "this parser is weak".
  • I3. Running every transfer pair in both directions shows whether a corpus is a good source, a good target, or both. Asymmetries like this tell you which data to borrow from. — For Mova (learn): Train-on-A/test-on-B in both directions across treebanks or genres would show which sources are worth adding to the silver-data pool.
  • I4. Pooling several foreign sources beat the best single source, which suggests that diversity in training data offsets the mismatch of any one source. — For Mova (learn): When building silver data for a thin domain such as fables or word problems, mixing several out-of-domain text sources is likely safer than relying on the single closest one.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.10814 · arXiv 2307.10814

@misc{retta2023cross,
  title = {{Cross-Corpus Multilingual Speech Emotion Recognition: Amharic vs. Other Languages}},
  author = {Retta, Ephrem Afele and Sutcliffe, Richard and Mahmood, Jabar and Berwo, Michael Abebe and Almekhlafi, Eiad and Khan, Sajjad Ahmed and Chaudhry, Shehzad Ashraf and Mhamed, Mustafa and Feng, Jun},
  year = {2023},
  doi = {10.48550/arxiv.2307.10814},
  eprint = {2307.10814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.10814}
}

References

50 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. M. Albornoz and D. H. Milone. Emotion recognition in never-seen languages using a novel ensemble method with emotion profiles. IEEE Transactions on Affective Computing, 8(1):43–53, 2015. find
  2. E. Almekhlafi, A.-M. Moeen, E. Zhang, J. Wang, and J. Peng. A classification benchmark for arabic alphabet phonemes with diacritics in deep neural networks. Computer Speech & Language, 71:101274, 2022. find
  3. M. Bhaykar, J. Yadav, and K. S. Rao. Speaker dependent, speaker independent and cross language emotion recognition from speech using gmm and hmm. In 2013 National conference on communications (NCC), pages 1–5. IEEE, 2013. find
  4. F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss. A database of german emotional speech. In Ninth European Conference on Speech Communication and Technology, 2005. find
  5. S. Deb and S. Dandapat. Multiscale amplitude feature and significance of enhanced vocal tract information for emotion classification. IEEE transactions on cybernetics, 49(3):802–815, 2018. find
  6. J. Deng, Z. Zhang, E. Marchi, and B. Schuller. Sparse autoencoder-based feature transfer learning for speech emotion recognition. In 2013 humaine association conference on affective computing and intelligent interaction, pages 511–516. IEEE, 2013. find
  7. N. Dey, A. S. Ashour, W. S. Mohamed, and N. G. Nguyen. Acoustic sensors in biomedical applications. In Acoustic sensors for biomedical applications, pages 43–47. Springer, 2019. find
  8. J. Duret, T. Parcollet, and Y. Estève. Learning multilingual expressive speech representation for prosody prediction without parallel data. In Speech Synthesis Workshop (SSW), 2023. find
  9. F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, et al. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE transactions on affective computing, 7(2):190–202, 2015. find
  10. G. Fairbanks and L. W. Hoaglin. An experimental study of the durational characteristics of the voice during the expression of emotion. Communications Monographs, 8(1):85–90, 1941. find
  11. P. Gangamohan, S. R. Kadiri, and B. Yegnanarayana. Analysis of emotional speech—a review. Toward Robotic Socially Believable Behaving Systems-Volume I, pages 205–238, 2016. find
  12. D. George, H. Shen, and E. Huerta. Deep transfer learning: A new deep learning glitch classification method for advanced ligo. arXiv preprint arXiv:1706.07446, 2017. ↗
  13. S. Goel and H. Beigi. Cross lingual cross corpus speech emotion recognition. arXiv preprint arXiv:2003.07996, 2020. ↗
  14. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. find
  15. D. Issa, M. F. Demirci, and A. Yazici. Speech emotion recognition with deep convolutional neural networks. Biomedical Signal Processing and Control, 59:101894, 2020. find
  16. R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain. Speech emotion recognition using deep learning techniques: A review. IEEE Access, 7:117327–117345, 2019. find
  17. M. U. Khan, A. R. Javed, M. Ihsan, and U. Tariq. A novel category detection of social media reviews in the restaurant industry. Multimedia Systems, pages 1–14, 2020. find
  18. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. find
  19. S. Kuchibhotla, H. D. Vankayalapati, and K. R. Anne. An optimal two stage feature selection for speech emotion recognition using acoustic features. International journal of speech technology, 19(4):657–667, 2016. find
  20. H. S. Kumbhar and S. U. Bhandari. Speech emotion recognition using mfcc features and lstm network. In 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), pages 1–3. IEEE, 2019. find
  21. S. Kwon et al. A cnn-assisted enhanced audio signal processing for speech emotion recognition. Sensors, 20(1):183, 2020. find
  22. S. Latif, J. Qadir, and M. Bilal. Unsupervised adversarial domain adaptation for cross-lingual speech emotion recognition. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 732–737. IEEE, 2019. find
  23. S. Latif, A. Qayyum, M. Usman, and J. Qadir. Cross lingual speech emotion recognition: Urdu vs. western languages. In 2018 International Conference on Frontiers of Information Technology (FIT), pages 88–93. IEEE, 2018. find
  24. S. Latif, R. Rana, S. Younis, J. Qadir, and J. Epps. Cross corpus speech emotion classification-an effective transfer learning technique. arXiv preprint arXiv:1801.06353, 2018. ↗
  25. I. Lefter, L. J. Rothkrantz, P. Wiggers, and D. A. Van Leeuwen. Emotion recognition from speech by combining databases and fusion of classifiers. In International Conference on Text, Speech and Dialogue, pages 353–360. Springer, 2010. find
  26. S. R. Livingstone and F. A. Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018. find
  27. A. Meftah, Y. Seddiq, Y. Alotaibi, and S.-A. Selouani. Cross-corpus arabic and english emotion recognition. In 2017 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), pages 377–381. IEEE, 2017. find
  28. A. D. Mengistu and M. A. Bedane. Text independent amharic language dialect recognition using neuro-fuzzy gaussian membership function. International Journal of Advanced Studies in Computers, Science and Engineering, 6(6):30, 2017. find
  29. P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016. ↗
  30. Z. Mossie and J.-H. Wang. Social network hate speech detection for amharic language. Computer Science & Information Technology, pages 41–55, 2018. find
  31. S. K. Pandey, H. S. Shekhawat, and S. Prasanna. Multi-cultural speech emotion recognition using language and speaker cues. Biomedical Signal Processing and Control, 83:104679, 2023. find
  32. E. A. Retta, E. Almekhlafi, R. Sutcliffe, M. Mhamed, H. Ali, and J. Feng. A new amharic speech emotion dataset and classification benchmark. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(1):1–22, 2023. find our card
  33. H. Sagha, P. Matejka, M. Gavryukova, F. Povolnỳ, E. Marchi, and B. W. Schuller. Enhancing multilingual recognition of emotion in speech by language identification. In Interspeech, pages 2949–2953, 2016. find
  34. K. Sailunaz, M. Dhaliwal, J. Rokne, and R. Alhajj. Emotion detection from text and speech: a survey. Social Network Analysis and Mining, 8(1):1–26, 2018. find
  35. M. Sajjad, S. Kwon, et al. Clustering-based speech emotion recognition by incorporating learned features and deep bilstm. IEEE Access, 8:79861–79875, 2020. find
  36. B. Schuller, A. Batliner, S. Steidl, and D. Seppi. Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech communication, 53(9-10):1062–1087, 2011. find
  37. B. Schuller, B. Vlasenko, F. Eyben, M. Wöllmer, A. Stuhlsatz, A. Wendemuth, and G. Rigoll. Cross-corpus acoustic emotion recognition: Variances and strategies. IEEE Transactions on Affective Computing, 1(2):119–131, 2010. find
  38. R. Sharmin, S. K. Rahut, and M. R. Huq. Bengali spoken digit classification: A deep learning approach using convolutional neural network. Procedia Computer Science, 171:1381–1388, 2020. find
  39. A. Shaw, R. K. Vardhan, and S. Saxena. Emotion recognition and classification in speech using artificial neural networks. International Journal of Computer Applications, 145(8):5–9, 2016. find
  40. A. S. Shinde and V. V. Patil. Speech emotion recognition system: A review. Available at SSRN 3869462, 2021. find
  41. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. ↗ our card
  42. C. Stanislavski. An actor prepares (new york. Theatre Arts, page 38, 1936. find
  43. M. Swain, S. Sahoo, A. Routray, P. Kabisatpathy, and J. N. Kundu. Study of feature combination using hmm and svm for multilingual odiya speech emotion recognition. International Journal of Speech Technology, 18(3):387– 393, 2015. find
  44. D. Wang and T. F. Zheng. Transfer learning for speech and language processing. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pages 1225–1237. IEEE, 2015. find
  45. K. Wang, G. Su, L. Liu, and S. Wang. Wavelet packet analysis for speaker-independent emotion recognition. Neurocomputing, 398:257–264, 2020. find
  46. Z. Xiao, D. Wu, X. Zhang, and Z. Tao. Speech emotion recognition cross language families: Mandarin vs. western languages. In 2016 International Conference on Progress in Informatics and Computing (PIC), pages 253–257. IEEE, 2016. find
  47. W. Zehra, A. R. Javed, Z. Jalil, H. U. Khan, and T. R. Gadekallu. Cross corpus multi-lingual speech emotion recognition using ensemble learning. Complex & Intelligent Systems, pages 1–10, 2021. find
  48. B. Zhang, E. M. Provost, and G. Essl. Cross-corpus acoustic emotion recognition from singing and speaking: A multi-task learning approach. In 2016 IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 5805–5809. IEEE, 2016. find
  49. Z. Zhang, F. Weninger, M. Wöllmer, and B. Schuller. Unsupervised learning in cross-corpus acoustic emotion recognition. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 523–528. IEEE, 2011. find
  50. K. Zvarevashe and O. Olugbara. Ensemble learning of hybrid acoustic features for speech emotion recognition. Algorithms, 13(3):70, 2020. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/retta-2023-cross-corpus-multilingual-speech-emotion/