Elamur Research⌕
☰
home / papers / 2020 / Efficient Arabic emotion recognition using deep neural networks

Efficient Arabic emotion recognition using deep neural networks

Summary

This paper by Yasser Hifny and Ahmed Ali tackles speech emotion recognition for Arabic, a language with few labelled emotional speech resources. They use the KSUEmotions corpus of Modern Standard Arabic and compare two end-to-end classifiers that take 13 MFCC frames as input. The first is a deep CNN, which they validate as a strong baseline on IEMOCAP. The second is a proposed hybrid. In the hybrid, convolutional layers first extract local acoustic features. Two bidirectional LSTM layers then model how the signal unfolds over time. An attention layer collapses the sequence into one weighted summary vector, which passes through a dense layer to a softmax over emotion classes. On the merged six-class corpus, the hybrid beats the CNN by about 2.2 points of accuracy on average across five folds. The CNN, however, is much faster to train and to run. The work gives an early reference point for Arabic speech emotion recognition and releases code for reuse.

Key points

  • The data is KSUEmotions (MSA, 23 speakers from Yemen, Saudi Arabia and Syria), recorded in two phases. Phase-1 has about 3.0 h with five emotions, including questioning. Phase-2 has about 2.2 h and replaces questioning with anger. The merged Phase-mix set has 5.2 h and six classes.
  • Features are 13 MFCCs extracted with the Essentia toolkit (25 ms frames, 10 ms shift, 26 Mel filter-banks). Models were built in Keras/TensorFlow and trained with Adam (learning rate 0.001, batch 32), with zero-padding up to 15.5 s.
  • The deep CNN baseline reaches 56.0% on four-class IEMOCAP, which is close to published state of the art. This justified using it as the reference system for the Arabic experiments.
  • On Phase-mix, the attention CNN-BLSTM-DNN averages 87.2% accuracy versus 85.0% for the deep CNN, and it wins in every fold. Replacing the attention layer with the last LSTM state lowers accuracy by more than 2 points absolute.
  • In the merged set, questioning and anger score worst, which the authors attribute to their smaller share of the data. The code is released at github.com/qcri/deepemotion.

Interesting ideas

  • I1. Turning a variable-length sequence into one vector with learned attention weights works better than taking the final recurrent state. The weights also show which positions drove the decision. — For Mova (frames): When classifying a whole clause or utterance (e.g. its speech act or an irony flag), Mova could pool over tree nodes with inspectable weights, so the explanation names the tokens that mattered.
  • I2. Before running a new architecture on a new, low-resource dataset, the authors first checked that their baseline matched known results on a standard benchmark. That anchors every later comparison. — For Mova (learn): Each new parser or tagger setup could first be shown to reproduce a known score on a reference UD treebank before its numbers on silver or new-domain data are trusted.
  • I3. Merging two corpora with partly different label sets produced a bigger dataset, but the classes found in only one part became the weakest. Per-class precision, recall and F1 exposed this, while overall accuracy hid it. — For Mova (learn): When pooling treebanks or silver sets converted from different UD dialects, Mova should report per-label LAS or F1, so that rare relations from only one source are not masked.
  • I4. The paper reports the accuracy gain together with its cost: the hybrid is more accurate, but the plain CNN is much faster. This makes the choice an explicit trade-off rather than a single winner. — For Mova (tree): Comparisons between the perceptron parser and gparse could always be logged as accuracy-versus-speed pairs, helping decide which parser serves which pipeline stage.
  • I5. An ablation that swaps out exactly one component (attention versus last state) isolates where the gain comes from. — For Mova (learn): Single-component swaps, such as removing one feature group or one repair rule, are a cheap negative-control style check that a claimed improvement really comes from the intended part.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2011.00346 · arXiv 2011.00346

@misc{ali2020efficient,
  title = {{Efficient Arabic emotion recognition using deep neural networks}},
  author = {Ali, Ahmed and Hifny, Yasser},
  year = {2020},
  doi = {10.48550/arxiv.2011.00346},
  eprint = {2011.00346},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2011.00346}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christos-Nikolaos Anagnostopoulos, Theodoros Iliou, and Ioannis Giannoukos, “Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,” Artificial Intelligence Review, vol. 43, no. 2, pp. 155–177, 2015. find
  2. Björn W Schuller, “Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends,” Communications of the ACM, vol. 61, no. 5, pp. 90–99, 2018. find
  3. Michael Neumann and Ngoc Thang Vu, “Attentive convolutional neural network based speech emotion recognition: A study on the impact of input features, signal length, and acted speech,” arXiv preprint arXiv:1706.00612, 2017. ↗
  4. Mousmita Sarma, Pegah Ghahremani, Daniel Povey, Nagendra Kumar Goel, Kandarpa Kumar Sarma, and Najim Dehak, “Emotion identification from raw speech signals using dnns,” Proc. Interspeech 2018, pp. 3097– 3101, 2018. find
  5. Yuanyuan Zhang, Jun Du, Zirui Wang, and Jianshu Zhang, “Attention based fully convolutional network for speech emotion recognition,” arXiv preprint arXiv:1806.01506, 2018. ↗
  6. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, no. 4, pp. 335, 2008. find
  7. Sepp Hochreiter and Jürgen Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  8. George Trigeorgis, Fabien Ringeval, Raymond Brueckner, Erik Marchi, Mihalis A Nicolaou, Björn Schuller, and Stefanos Zafeiriou, “Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 5200–5204. find
  9. Tara N Sainath, Oriol Vinyals, Andrew Senior, and Haşim Sak, “Convolutional, long short-term memory, fully connected deep neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4580–4584. find
  10. Aharon Satt, Shai Rozenberg, and Ron Hoory, “Efficient emotion recognition from speech using deep learning on spectrograms,” Proc. Interspeech 2017, pp. 1089–1093, 2017. find
  11. Myungsu Chae, Tae-Ho Kim, Young Hoon Shin, Jun-Woo Kim, and Soo-Young Lee, “End-to-end multimodal emotion and gender recognition with dynamic weights of joint loss,” arXiv preprint arXiv:1809.00758, 2018. ↗
  12. Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, and Gerald Penn, “Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. IEEE, 2012, pp. 4277–4280. find
  13. Alex Graves, Santiago Fernández, and Jürgen Schmidhuber, “Bidirectional lstm networks for improved phoneme classification and recognition,” in International Conference on Artificial Neural Networks. Springer, 2005, pp. 799–804. find
  14. Ali Meftah, Yousef A Alotaibi, and Sid-Ahmed Selouani, “Designing, building, and analyzing an arabic speech emotional corpus: Phase 2,” in 5th International Conference on Arabic Language Processing, 2014, pp. 181–184. find
  15. Dmitry Bogdanov, Nicolas Wack, Emilia Gómez Gutiérrez, Sankalp Gulati, Perfecto Herrera Boyer, Oscar Mayor, Gerard Roma Trepat, Justin Salamon, José Ricardo Zapata González, and Xavier Serra, “Essentia: An audio analysis library for music information retrieval,” in Britto A, Gouyon F, Dixon S, editors. 14th Conference of the International Society for Music Information Retrieval (ISMIR); 2013 Nov 4-8; Curitiba, Brazil.[place unknown]: ISMIR; 2013. p. 493-8. International Society for Music Information Retrieval (ISMIR), 2013. find
  16. François Chollet et al., “Keras,” 2015. find
  17. Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2020-efficient-arabic-emotion-recognition-using/