Elamur Research⌕
☰
home / papers / 2022 / A New Amharic Speech Emotion Dataset and Classification Benchmark

A New Amharic Speech Emotion Dataset and Classification Benchmark

Summary

Amharic, the national language of Ethiopia, has had no corpus for speech emotion recognition (SER). The authors built ASED, which they describe as the first such resource. Sixty-five native speakers recorded short emotional utterances in the four main dialects, and dialect-matched judges validated the emotion labels. The dataset is freely downloadable. On top of it the authors propose VGGb, a slimmed VGG network with four convolutional layers, and run three experiments.

The first experiment compares Mel-spectrogram and MFCC input features. It uses ordinary cross-validation plus three splits that each hold out one factor: sentences, dialect or speaker group. The second experiment benchmarks VGGb against ResNet50, Alex-Net and LSTM. In the third, VGGb is trained not only on ASED but also on two corpora in other languages, RAVDESS (English) and EMO-DB (German).

The main findings:

  • MFCC beats Mel-spectrograms under every split.
  • VGGb nearly matches ResNet50 in accuracy and trains far faster.
  • Accuracy across the three languages stays within a narrow band.

Together these results give low-resource Amharic a usable SER baseline and a reference dataset.

Key points

  • ASED contains 2,474 clips of two to four seconds from 65 speakers. They cover Gojjam, Wollo, Shewa and Gonder dialects and five emotions: neutral 522, fearful 510, happy 486, sad 470, angry 486.
  • Eight judges labelled the clips, two per dialect, and their agreement reached Fleiss kappa = 0.8.
  • In the speaker-group hold-out (professional, semi-professional, amateur), MFCC averaged 76.91% accuracy against 71.57% for Mel-spectrograms. MFCC also varied less across folds (std. dev. 1.53 vs 2.50).
  • Model comparison on ASED with MFCC:
    • ResNet50: 91.13% accuracy, about 8h45m training.
    • VGGb: 90.73% accuracy, about 31 min training.
    • Alex-Net: 81.82%.
    • LSTM: 66.94%.
  • VGGb's accuracy by corpus was 86.15% on RAVDESS, 88.71% on EMO-DB and 90.73% on ASED. All features were extracted with librosa v0.7.2, and results are averaged over five runs.

Interesting ideas

  • I1. The authors hold out one factor at a time (a sentence set, a dialect, a speaker group) to check that a design choice wins regardless of who spoke what; this separates true superiority from memorised surface material. — For Mova (learn): Evaluation could hold out whole genres, source treebanks or text authors when deciding between two pipeline variants, and keep only changes that win on every held-out slice.
  • I2. They report the spread across folds, not just the mean, and treat lower variance as an extra sign that one option is more robust. — For Mova (learn): Repair rules and teacher tiers could be ranked by both mean gain and its standard deviation across held-out slices, so that brittle rules are flagged early.
  • I3. Annotators are matched to the dialect they judge, and a chance-corrected agreement score certifies the labels before any model sees them. — For Mova (learn): Silver data agreed by several witness parsers could carry a Fleiss-kappa-style agreement score per sentence or per relation type, so that low-agreement material is filtered before training.
  • I4. Accuracy is weighed together with training cost: a model 0.4 points worse but many times faster is preferred as the practical default. — For Mova (tree): Comparisons between the perceptron parser and gparse could report LAS alongside training and inference time, which makes the trade-off explicit when choosing a default parser.
  • I5. The same architecture and settings are run unchanged on corpora in several languages, to show that results come from the method rather than one dataset. — For Mova (tree): After UD dialect conversion, one fixed parser configuration could be trained on several treebanks to check that gains do not depend on a single corpus's annotation habits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2201.02710 · arXiv 2201.02710

@misc{retta2022new,
  title = {{A New Amharic Speech Emotion Dataset and Classification Benchmark}},
  author = {Retta, Ephrem A. and Almekhlafi, Eiad and Sutcliffe, Richard and Mhamed, Mustafa and Ali, Haider and Feng, Jun},
  year = {2022},
  doi = {10.48550/arxiv.2201.02710},
  eprint = {2201.02710},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.02710}
}

References

36 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Anberbir, M. Gasser, T. Takara, and K. D. Yoon. Grapheme-to-phoneme conversion for amharic text-to-speech system. In Proceedings of conference on human language technology for development, Bibliotheca Alexandrina, Alexandria, 2011. find
  2. audacity. audacity, 2020. [Online; accessed 31-October-2020]. find
  3. S. Basu, A. Bag, M. Aftabuddin, M. Mahadevappa, J. Mukherjee, and R. Guha. Effects of emotion on physiological signals. In 2016 IEEE Annual India Conference (INDICON), pages 1–6. IEEE, 2016. find
  4. B. Belay, T. Habtegebrial, M. Meshesha, M. Liwicki, G. Belay, and D. Stricker. Amharic ocr: An end-to-end learning. Applied Sciences, 10(3):1117, 2020. find
  5. F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss. A database of german emotional speech. In Ninth European Conference on Speech Communication and Technology, 2005. find
  6. C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335– 359, 2008. find
  7. K. Dupuis and M. K. Pichora-Fuller. Recognition of emotional speech for younger and older talkers: behavioural findings from the toronto emotional speech set. Canadian Acoustics, 39(3):182–183, 2011. find
  8. D. George, H. Shen, and E. Huerta. Deep transfer learning: A new deep learning glitch classification method for advanced ligo. arXiv preprint arXiv:1706.07446, 2017. ↗
  9. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. find
  10. D. Issa, M. F. Demirci, and A. Yazici. Speech emotion recognition with deep convolutional neural networks. Biomedical Signal Processing and Control, 59:101894, 2020. find
  11. P. Jackson and S. Haq. Surrey audio-visual expressed emotion (savee) database. University of Surrey: Guildford, UK, 2014. find
  12. L. Kerkeni, Y. Serrestou, M. Mbarki, K. Raoof, M. A. Mahjoub, and C. Cleder. Automatic speech emotion recognition using machine learning. In Social Media and Machine Learning. IntechOpen, 2019. find
  13. R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain. Speech emotion recognition using deep learning techniques: A review. IEEE Access, 7:117327–117345, 2019. find
  14. S. G. Koolagudi, R. Reddy, J. Yadav, and K. S. Rao. Iitkgp-sehsc: Hindi speech corpus for emotion analysis. In 2011 International conference on devices and communications (ICDeCom), pages 1–5. IEEE, 2011. find
  15. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. find
  16. H. S. Kumbhar and S. U. Bhandari. Speech emotion recognition using mfcc features and lstm network. In 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), pages 1–3. IEEE, 2019. find
  17. S. Kwon et al. Clstm: Deep feature-based speech emotion recognition using the hierarchical convlstm network. Mathematics, 8(12):2133, 2020. find
  18. S. Kwon et al. A cnn-assisted enhanced audio signal processing for speech emotion recognition. Sensors, 20(1):183, 2020. find
  19. Y. Li, J. Tao, L. Chao, W. Bao, and Y. Liu. Cheavd: a chinese natural emotional audio–visual database. Journal of Ambient Intelligence and Humanized Computing, 8(6):913–924, 2017. find
  20. S. R. Livingstone and F. A. Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018. find
  21. A. D. Mengistu and M. A. Bedane. Text independent amharic language dialect recognition using neuro-fuzzy gaussian membership function. International Journal of Advanced Studies in Computers, Science and Engineering, 6(6):30, 2017. find
  22. P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016. ↗
  23. Z. Mossie and J.-H. Wang. Social network hate speech detection for amharic language. Computer Science & Information Technology, pages 41–55, 2018. find
  24. W. Mulugeta and M. Gasser. Learning morphological rules for amharic verbs using inductive logic programming. Language Technology for Normalisation of Less-Resourced Languages, 7, 2012. find
  25. M. K. Pichora-Fuller and K. Dupuis. Toronto emotional speech set (TESS), 2010. find
  26. J. J. Randolph. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss’ fixed-marginal multirater kappa. Online submission, 2005. find
  27. M. Sajjad, S. Kwon, et al. Clustering-based speech emotion recognition by incorporating learned features and deep bilstm. IEEE Access, 8:79861–79875, 2020. find
  28. B. W. Schuller. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM, 61(5):90–99, 2018. find
  29. R. Sharmin, S. K. Rahut, and M. R. Huq. Bengali spoken digit classification: A deep learning approach using convolutional neural network. Procedia Computer Science, 171:1381–1388, 2020. find
  30. A. Shaw, R. K. Vardhan, and S. Saxena. Emotion recognition and classification in speech using artificial neural networks. International Journal of Computer Applications, 145(8):5–9, 2016. find
  31. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. ↗ our card
  32. C. Stanislavski. An actor prepares (new york. Theatre Arts, page 38, 1936. find
  33. R. N. Tak, D. M. Agrawal, and H. A. Patil. Novel phase encoded mel filterbank energies for environmental sound classification. In International Conference on Pattern Recognition and Machine Intelligence, pages 317–325. Springer, 2017. find
  34. V. P. Tank and S. Hadia. Creation of speech corpus for emotion analysis in gujarati language and its evaluation by various speech parameters. International Journal of Electrical and Computer Engineering, 10(5):4752, 2020. find
  35. G. S. Tomas. Speech Emotion Recognition Using Convolutional Neural Networks. PhD thesis, Institute of Language and Communication, Technical University of Berlin, 2019. find
  36. N. Vryzas, R. Kotsakis, A. Liatsou, C. A. Dimoulas, and G. Kalliris. Speech emotion recognition for performance interaction. Journal of the Audio Engineering Society, 66(6):457–467, 2018. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/retta-2022-new-amharic-speech-emotion-dataset/