Elamur Research⌕
☰
home / papers / 2026 / Hybrid CNN-Transformer Architecture for Arabic Speech Emotion Recognition

Hybrid CNN-Transformer Architecture for Arabic Speech Emotion Recognition

Summary

The paper tackles Speech Emotion Recognition (SER) for Arabic, where annotated emotional speech data is scarce and most past work used shallow classifiers or plain CNNs. The authors propose a hybrid model that takes log-scaled Mel-spectrograms as input. Stacked convolution and pooling layers learn local spectral patterns. Transformer encoders with multi-head self-attention and sinusoidal positional encoding then model dependencies across the whole utterance. Global average pooling and fully connected layers with softmax produce the emotion label.

The audio is preprocessed before feature extraction:

  • resampled to 16 kHz and converted to mono;
  • normalized to zero mean and unit variance;
  • cleaned of silence, overlapping voices and noisy segments.

The system is evaluated on the EYASE corpus of semi-natural Egyptian Arabic. It reaches 97.8% accuracy and a macro F1 of 0.98, which the authors present as state of the art and as a reference point for future Arabic SER research. The work suggests that attention-based models can be effective for low-resource languages even with limited corpora.

Key points

  • Data: EYASE (Egyptian Arabic speech emotion), a publicly available semi-natural corpus. The related-work table also covers KSUEmotions (acted) and a TV-broadcast natural-speech set.
  • Features: 25 ms Hamming windows with a 10 ms shift, then STFT, a 128-filter Mel bank and a log transform.
  • Architecture: CNN feature extractor, Transformer encoder (scaled dot-product, multi-head attention, sinusoidal positions), global average pooling, then dense layers with softmax.
  • Result: 97.8% accuracy and macro F1 of 0.98 on EYASE.
  • Positioning: few earlier studies combined CNNs with Transformers for Arabic SER. Prior Arabic work relied on SVM, KNN, SMO, deep CNNs or CNN+BLSTM with attention.

Interesting ideas

  • I1. Split the work between a local pattern extractor (convolutions) and a global context model (self-attention), so each component handles the scale it is best at. — For Mova (tree): gparse could run a small character-level convolution over word forms to capture affixes and shape before its transformer layers, giving better signals for rare or unseen words.
  • I2. Compress the input onto a perceptually motivated, log-scaled axis to tame dynamic range and stabilize training. — For Mova (learn): count-like features such as word frequency or distance could be log-bucketed before entering the perceptron or gparse, so a few extreme values do not dominate the weights.
  • I3. Clean the data before extracting features by discarding segments with overlap, silence or noise, rather than hoping the model learns to ignore them. — For Mova (learn): silver sentences where witness parsers disagree, or which contain garbled extraction artifacts, could be dropped before training instead of being down-weighted afterwards.
  • I4. Report macro F1 alongside accuracy, so that performance on minority classes is visible and not hidden by frequent ones. — For Mova (learn): parser and frame evaluations could publish macro-averaged scores over deprels and frame types, which exposes weak rare labels like vocative or rare VerbNet roles.
  • I5. Classify every prior dataset by how natural its material is (acted, semi-natural or natural) when comparing methods. — For Mova (answer): test suites could be tagged as synthetic, edited or natural text, so that scores from easy constructed items are not mistaken for robustness on real fables and school texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.07357 · arXiv 2604.07357

@misc{gheffari2026hybrid,
  title = {{Hybrid CNN-Transformer Architecture for Arabic Speech Emotion Recognition}},
  author = {Gheffari, Youcef Soufiane and Benouddane, Oussama Mustapha and Silarbi, Samiya},
  year = {2026},
  doi = {10.48550/arxiv.2604.07357},
  eprint = {2604.07357},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.07357}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. B. Akçay and K. Oğuz, “Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers,” Speech Communication, vol. 116, pp. 56–76, 2020. find
  2. United Nations, “Official Languages,” 2023. [Online]. Available: https://www.un.org/en/our-work/official-languages ↗
  3. A. Dahou, A. H. H. Dahou, M. A. Cheragui, A. Abdedaiem, M. A. A. Al-qaness, M. Abd Elaziz, A. A. Ewees, and Z. Zhonglong, “A survey on dialect Arabic processing and analysis: Recent advances and future trends,” ACM Trans. Asian Low-Resour. Lang. Inf. Process., 2025. find
  4. Z. Zhang, F. Weninger, M. Wöllmer, and B. Schuller, “Unsupervised learning in cross-corpus acoustic emotion recognition,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2011, pp. 2049– 2053. find
  5. B. Schuller, S. Steidl, A. Batliner, et al., “The INTERSPEECH 2010 Paralinguistic Challenge,” in Proc. INTERSPEECH, 2010, pp. 2794– 2797. find
  6. Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, “Electron spectroscopy studies on magneto-optical media and plastic substrate interface,” IEEE Transl. J. Magn. Japan, vol. 2, pp. 740–741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982]. find
  7. Y. Kim, H. Lee, and E. M. Provost, “Deep learning for robust feature generation in audiovisual emotion recognition,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2013, pp. 3687–3691. find
  8. S. Latif, “Deep representation learning for speech emotion recognition,” Ph.D. dissertation, University of Southern Queensland, 2022. find
  9. T. Meng, Y. Shou, W. Ai, N. Yin, and K. Li, “Deep imbalanced learning for multimodal emotion recognition in conversations,” IEEE Transactions on Artificial Intelligence, 2024. find
  10. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 5998–6008. find
  11. A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, et al., “Conformer: Convolution-augmented transformers for speech recognition,” in Proc. INTERSPEECH, 2020, pp. 5036–5040. find
  12. O. Mahmoudi and M. F. Bouami, “Arabic speech emotion recognition using deep neural network,” in *Proc. Int. Conf. on Digital Technologies and Applications*, 2023, pp. 124–133. find
  13. W. Ismaiel, A. Alhalangy, A. O. Y. Mohamed, and A. I. A. Musa, “Deep learning, ensemble and supervised machine learning for Arabic speech emotion recognition,” Eng. Technol. Appl. Sci. Res., vol. 14, no. 2, pp. 13757–13764, 2024. find
  14. Y. Hifny and A. Ali, “Efficient Arabic emotion recognition using deep neural networks,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2019, pp. 6710–6714. find our card
  15. T. L. Nwe, S. W. Foo, and L. C. De Silva, “Speech emotion recognition using hidden Markov models,” Speech Commun., vol. 41, no. 4, pp. 603–623, 2003. find
  16. S. PS and G. Mahalakshmi, “Emotion models: a review,” Int. J. Control Theory Appl., vol. 10, no. 8, pp. 651–657, 2017. find
  17. F. Eyben, M. Wöllmer, and B. Schuller, “OpenSMILE – The Munich versatile and fast open-source audio feature extractor,” in Proc. ACM Multimedia, 2010, pp. 1459–1462. find
  18. T. N. Sainath, R. J. Weiss, A. Senior, K. W. Wilson, and O. Vinyals, “Learning the speech front-end with raw waveform CLDNNs,” in Proc. INTERSPEECH, 2015, pp. 1–5. find
  19. B. Schuller, S. Steidl, A. Batliner, et al., “The INTERSPEECH 2013 Computational Paralinguistics Challenge: Social signals, conflict, emotion, autism,” in Proc. INTERSPEECH, 2013, pp. 148–152. find
  20. L. R. Rabiner and R. W. Schafer, *Theory and Applications of Digital Speech Processing*. Upper Saddle River, NJ, USA: Pearson, 2010. find
  21. L. D. Alsteris and K. K. Paliwal, “Short-time phase spectrum in speech processing: A review and some experimental results,” Digit. Signal Process., vol. 17, no. 3, pp. 578–616, 2007. find
  22. T. Haustein, A. Forck, H. Gäbler, V. Jungnickel, and S. Schiffermüller, “Real-time signal processing for multiantenna systems: algorithms, optimization, and implementation on an experimental test-bed,” EURASIP Journal on Advances in Signal Processing, vol. 2006, no. 1, p. 027573, 2006. find
  23. S. Stevens, J. Volkmann, and E. Newman, “A scale for the measurement of the psychological magnitude pitch,” J. Acoust. Soc. Am., vol. 8, no. 3, pp. 185–190, 1937. find
  24. A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2013, pp. 6645–6649. find
  25. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2020. find our card
  26. T. N. Sainath, B. Kingsbury, A. Mohamed, G. E. Dahl, G. Saon, H. Soltau, T. Beran, A. Y. Aravkin, and B. Ramabhadran, “Improvements to deep convolutional neural networks for LVCSR,” in Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2013, pp. 315–320. find
  27. M. Radfar, R. Barnwal, R. V. Swaminathan, F.-J. Chang, G. P. Strimel, N. Susanj, and A. Mouchtaris, “Convrnn-t: Convolutional augmented recurrent neural network transducers for streaming speech recognition,” arXiv preprint arXiv:2209.14868, 2022. ↗
  28. K. Miyazaki, T. Komatsu, T. Hayashi, S. Watanabe, T. Toda, and K. Takeda, “Convolution-augmented transformer for semi-supervised sound event detection,” in Proc. Workshop Detection Classification Acoust. Scenes Events (DCASE), 2020, pp. 100–104. find
  29. L. Abdel-Hamid, “Egyptian Arabic speech emotion recognition using prosodic, spectral, and wavelet features,” Speech Communication, vol. 122, pp. 19–35, 2020. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gheffari-2026-hybrid-cnn-transformer-architecture-arabic/