Elamur Research⌕
☰
home / papers / 2025 / MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages

MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages

Paper · 2025 · Hardik B. Sailor, Aw Ai Ti, Chen Fang Yih Nancy et al.

Summary

The paper addresses speech emotion recognition (SER) for Singapore's main languages (English, Chinese, Malay, Tamil) and other Southeast Asian speech. In this setting, labelled data is scarce, annotators often disagree, and general Audio-LLMs tend to focus on transcribed content rather than prosody.

The authors build MERaLiON-SER on a frozen Whisper-Medium encoder with LoRA adapters in its attention layers. On top sits a downstream stack with multiscale and hierarchical attention pooling and modified ECAPA-TDNN blocks, which use GroupNorm in place of BatchNorm. The model has two heads:

  • one predicts seven emotion categories, trained with weighted cross-entropy;
  • the other regresses arousal, valence and dominance, trained with a CCC loss.

Training combines several sources:

  • pseudo-labelled in-house Singaporean speech (SG-ECMT);
  • human-labelled TV and movie clips (SGTV);
  • public corpora.

Robustness comes from MixUp, MUSAN noise and speed perturbation.

On a human-validated Singapore test set and on public benchmarks, MERaLiON-SER beats open speech encoders such as emotion2vec and large Audio-LLMs such as GPT-4o-Audio and Gemini-2.5-flash. The authors read this as evidence that targeted paralinguistic models still matter.

Key points

  • Seven classes: Neutral, Happy, Sad, Angry, Surprised, Fearful, Disgusted. Three continuous scores (arousal, valence, dominance) come from a sigmoid head. The loss weights are λcat = 1.0 and λdim = 0.5, with label smoothing of 0.1.
  • SG-ECMT training data was pseudo-labelled in 4 s windows with emotion2vec. It contains 27,458 English, 14,212 Chinese, 14,370 Malay and 10,169 Tamil samples. SGTV has about 117k human-labelled clips (about 120 h) of English, Mandarin and code-mixed speech. Public training corpora are CREMA-D, M3ED, ESD and MELD.
  • The SG-ECMT evaluation set has 1,880 clips of 10–30 s. Three native speakers per language corrected the automatic labels, and the final labels were set by majority vote. Public tests are MSP-podcast test1, IEMOCAP, MELD, M3ED and IndoWaveSentiment.
  • Results:
    • UAR is about 64–70% on English corpora and about 57–60% on Chinese and Indonesian data.
    • The best emotion2vec variant (seed) trails by 4–6 points absolute.
    • GPT-4o-Audio and Gemini-2.5-flash trail by 8–12 points.
    • The much larger MERaLiON-2-10B also scores below MERaLiON-SER.
  • Final predictions average the four checkpoints with the lowest development categorical loss.

Interesting ideas

  • I1. Agreement filter for pseudo-labels: an emotional label was kept only when two emotion2vec+ variants predicted the same emotion; segments without agreement fell back to neutral, so automatic labels were trusted only under consensus.
  • I2. Two-pass labelling: the first pseudo-labels trained a model, which then relabelled the same corpus, turning a teacher-derived dataset into a self-refined one.
  • I3. Discrete plus dimensional outputs in one model: pairing category labels with graded scores gives a coarse decision plus continuous nuance, and the scores still carry meaning for cases outside the fixed label set.
  • I4. Cheap labels for training, human checks for testing: training relied on noisy automatic labels, but the test set was corrected by three native speakers with majority voting, so the reported score does not inherit the labeller's bias.
  • I5. Benchmark the specialist against big generalists: the paper shows a compact task-specific model beating far larger Audio-LLMs, a useful check on whether scale really helps a narrow skill.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver UD trees could keep a labelled arc only where the witness parsers agree, and route disagreements to a conservative default or to LLM-teacher review.
  • I2 (learn): after training on witness-agreed silver data, gparse could re-annotate that data, and the diffs could seed self-induced repair rules.
  • I3 (frames): speech-act or irony detection could output a discrete act together with graded scales such as intensity or speaker certainty, which keeps borderline cases explainable.
  • I4 (learn): keep a small human-adjudicated gold set separate from silver data, so silver-trained parser gains are measured honestly.
  • I5 (answer): compare Mova's explainable readers on fable QA and word problems directly with the LLM teachers, to show where structure beats size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSpeech & spoken language · 480 works

Cite

Cite as Hardik B. Sailor, Aw Ai Ti, Chen Fang Yih Nancy, Chiu Ying Lay, Ding Yang, He Yingxu, Jiang Ridong, Li Jingtao, Liao Jingyi, Liu Zhuohan, Lu Yanfeng, Ma Yi, Manas Gupta, Muhammad Huzaifah Bin Md Shahrin, Nabilah Binte Md Johan, Nattadaporn Lertcheva, Pan Chunlei, Pham Minh Duc, Siti Maryam Binte Ahmad Subaidi, Siti Umairah Binte Mohammad Salleh, Sun Shuo, Tarun Kumar Vangani, Wang Qiongqiong, Won Cheng Yi Lewis, Wong Heng Meng Jeremy, Wu Jinyang, Zhang Huayun, Zhang Longyin and Zou Xunlong. 2025. MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages. arXiv:2511.04914.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.04914 · arXiv 2511.04914

@misc{sailor2025meralion,
  title = {{MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages}},
  author = {Hardik B. Sailor, Aw Ai Ti, Chen Fang Yih Nancy, Chiu Ying Lay, Ding Yang, He Yingxu, Jiang Ridong, Li Jingtao, Liao Jingyi, Liu Zhuohan, Lu Yanfeng, Ma Yi, Manas Gupta, Muhammad Huzaifah {Bin Md Shahrin}, Nabilah Binte Md Johan, Nattadaporn Lertcheva, Pan Chunlei, Pham Minh Duc, Siti Maryam Binte Ahmad Subaidi, Siti Umairah Binte Mohammad Salleh, Sun Shuo, Tarun Kumar Vangani, Wang Qiongqiong, Won Cheng Yi Lewis, Wong Heng Meng Jeremy, Wu Jinyang, Zhang Huayun, Zhang Longyin and Zou Xunlong},
  year = {2025},
  doi = {10.48550/arxiv.2511.04914},
  eprint = {2511.04914},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.04914}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiama, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, and et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. ↗
  2. R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, and et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2025. ↗
  3. C. Busso, M. Bulut, C.-C. Lee, E. Kazemzadeh, E. M. Provost, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335–359, 2008. ↗
  4. C. Busso, M. Al-Hames, and Z. Liu. MSP-Podcast: A large-scale corpus for research on affective computing in the wild. IEEE Transactions on Affective Computing, 8(4):455–469, 2017. find
  5. H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma. CREMA-D: Crowdsourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4): 377–390, 2014. find
  6. J. Chen, Z. Guo, J. Chun, P. Wang, A. Perrault, and M. Elsner. Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance. arXiv preprint arXiv:2510.10444, 2025. ↗
  7. B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. In Proc. Interspeech, pages 3830–3834, 2020. find
  8. T. Feng, J. Lee, A. Xu, Y. Lee, T. Lertpetchpun, X. Shi, H. Wang, T. Thebaud, L. Moro-Velazquez, D. Byrd, et al. Vox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits. arXiv preprint arXiv:2505.14648, 2025. ↗
  9. Y. He, Z. Liu, S. Sun, B. Wang, W. Zhang, X. Zou, N. F. Chen, and A. T. Aw. MERaLiON-AudioLLM: Technical report. arXiv preprint arXiv:2412.09818, 2024. ↗
  10. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. LoRA: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
  11. Z. Huang, H. Yang, M. Yan, and Q. Yang. M3ED: A multi-modal multi-turn multi-level multi-label emotional dialogue dataset. In Proceedings of the 29th ACM International Conference on Multimedia (MM ’21), 2021. find
  12. W. Kang, J. Jia, C. Wu, W. Zhou, E. Lakomkin, Y. Gaur, L. Sari, S. Kim, K. Li, J. Mahadeokar, et al. Frozen large language models can perceive paralinguistic aspects of speech. arXiv preprint arXiv:2410.01162, 2024. ↗
  13. H. Kim, S. Seo, K. Jeong, O. Kwon, S. Kim, J. Kim, J. Lee, E. Song, M. Oh, J.-W. Ha, S. Yoon, and K. M. Yoo. Paralinguistics-aware speech-empowered large language models for natural conversation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
  14. G.-T. Lin, C.-H. Chiang, and H.-Y. Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786, 2024a. ↗
  15. W.-C. Lin, S. Ghaffarzadegan, L. Bondi, A. Kumar, S. Das, and H.-H. Wu. CLAP4Emo: ChatGPT-Assisted Speech Emotion Retrieval with Natural Language Supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11791–11795, 2024b. find
  16. C. Liu, M. Aljunied, G. Chen, H. P. Chan, W. Xu, Y. Rong, and W. Zhang. SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia. https://github.com/DAMO-NLP-SG/ SeaLLMs-Audio, 2025. ↗
  17. K.-H. Lu, Z. Chen, S.-W. Fu, H. Huang, B. Ginsburg, Y.-C. Wang, and H.-y. Lee. DeSTA: Enhancing speech language models through descriptive speech-text alignment. In Interspeech, pages 4159– 4163, 2024. doi: 10.21437/Interspeech.2024-457. ↗
  18. K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, J. Balam, B. Ginsburg, Y.-C. Wang, and H.-y. Lee. DeSTA2: Developing instruction-following speech language model without speech instruction-tuning data. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025. find
  19. Z. Ma, M. Chen, H. Zhang, Z. Zheng, W. Chen, X. Li, J. Ye, X. Chen, and T. Hain. EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark. In Interspeech 2024, pages 1580–1584, 2024a. find
  20. Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics (ACL), 2024b. find
  21. A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179–1210, 2022. ↗
  22. A. R. Naini, M. A. Kohler, E. Richerson, D. Robinson, and C. Busso. Generalization of self-supervised learning-based representations for cross-domain speech emotion recognition. In 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12031–12035, 2024. find
  23. M. Osman, D. Z. Kaplan, and T. Nadeem. SER Evals: In-domain and Out-of-domain benchmarking for speech emotion recognition. In Interspeech 2024, pages 1395–1399, 2024. doi: 10.21437/ Interspeech.2024-2440. find
  24. S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computation al Linguistics (ACL), 2019.
  25. D. Snyder, G. Chen, and D. Povey. MUSAN: A Music, Speech, and Noise Corpus. arXiv preprint arXiv:1510.08484, 2015. ↗
  26. C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang. BLSP-Emo: Towards empathetic large speech-language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. find
  27. Q. Wang, H. B. Sailor, T. Liu, and A. T. Aw. Contextual paralinguistic data creation for multi-modal speech-llm: Data condensation and spoken QA generation. In Proc. Interspeech, 2025a. find
  28. Q. Wang, H. B. Sailor, T. Liu, W. Zhang, M. Huzaifah, N. Lertcheva, S. Sun, N. F. Chen, J. Wu, and A. Aw. Benchmarking contextual and paralinguistic reasoning in speech-llms: A case study with in-the-wild data. In Findings of EMNLP 2025, 2025b. ↗
  29. H. Wu, H.-C. Chou, K.-W. Chang, L. Goncalves, J. Du, J.-S. R. Jang, C.-C. Lee, and H.-Y. Lee. Empower typed descriptions by large language models for speech emotion recognition. In Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1–6. IEEE, 2024. find
  30. Z. Wu, Z. Gong, L. Ai, P. Shi, K. Donbekci, and J. Hirschberg. Beyond silent letters: Amplifying LLMs in emotion recognition with vocal nuances. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2202–2218, 2025. find
  31. Y. Xu, H. Chen, J. Yu, Q. Huang, Z. Wu, S.-X. Zhang, G. Li, Y. Luo, and R. Gu. SECap: Speech emotion captioning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19323–19331, 2024. find
  32. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. Mixup: Beyond empirical risk minimization. ICLR, 2018. ↗
  33. Y. Zhou and L. Zhou. Emotional speech dataset (ESD) for voice conversion. In Interspeech, pages 2793–2797, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sailor-2025-meralion-ser-robust-speech-emotion/