Elamur Research⌕
☰
home / papers / 2025 / Multilingual Stutter Event Detection for English, German, and Mandarin Speech

Multilingual Stutter Event Detection for English, German, and Mandarin Speech

Paper · 2025 · Felix Haas and Sebastian P. Bayerl

Summary

Automatic stuttering detection has long been limited by small, single-language datasets, so systems often fail to generalize. This paper asks whether training one model on several languages, including one outside the Germanic family, helps or hurts. The authors combine four corpora with compatible clip-level labels: KSoF (German), SEP-28k-E and FluencyBank (English), and AS-70 (Mandarin). The labels cover five dysfluency types: block, interjection, prolongation, sound repetition and word repetition. They fine-tune a large wav2vec 2.0 model (XLSR-53 weights) as a multi-label classifier. Each class gets its own sigmoid output, the main task uses focal loss, and language identification serves as an auxiliary task. Adding Mandarin data did not degrade performance on the Germanic corpora. A length-limited trilingual model (TRILANG-LL) beat the previous state of the art on KSoF for four of five dysfluency types, although German made up under 8% of its training data. Word-repetition detection also seemed to benefit from more diverse data. The results suggest stuttering has cross-linguistic regularities that a language-agnostic detector can exploit.

Key points

  • Data: the four corpora are KSoF (5597 clips of 3 s from German therapy recordings), SEP-28k-E (about 28k English podcast clips), FluencyBank (about 4144 English clips) and AS-70 (Mandarin, variable-length clips averaging 4.12 s).
  • Model and loss:
    • The backbone is W2V2 large with XLSR-53 weights, chosen in preliminary runs over English-only and Chinese (WenetSpeech) weights.
    • The classification head uses mean pooling.
    • The loss is a weighted sum with w_main = 0.9, combining focal loss (α = 0.7, γ = 3) for the main task and BCE for the auxiliary language-ID task.
  • Training: AdamW, batch size 8, learning rate 3×10⁻⁵, up to 20 epochs, early stopping with patience 5.
  • Inference on long clips: AS-70 clips longer than 3 s are cut into 3 s windows with 1.5 s overlap, and the window predictions are merged back into one clip-level result.
  • Length limit: clips above 7 s (about 14% of AS-70) were excluded from training.
    • Such long clips tend to contain more than two dysfluencies on average.
    • The labels cannot express how often a type repeats within a clip, which makes these clips noisy for training.

Interesting ideas

  • I1. Using source-language identification as a cheap auxiliary head with a small loss weight regularizes a model trained on pooled heterogeneous corpora.
  • I2. Giving each label its own sigmoid output plus focal loss lets rare, co-occurring categories be learned without a softmax forcing a single winner.
  • I3. Overlapping fixed windows whose outputs are aggregated let a fixed-context model handle inputs of any length, at the cost of losing the exact location of an event.
  • I4. The authors deliberately froze an older architecture so that any gains could be credited to the new data rather than the model.
  • I5. Dropping training examples whose labels are too coarse to describe them (long clips with many unlabeled repeats) improved results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when gparse is trained on several merged treebanks, an auxiliary head predicting the source treebank or UD dialect could help the shared encoder keep annotation-style differences apart.
  • I2 (frames): speech acts, irony and implicature cues often co-occur and are rare, so independent per-label outputs with focal weighting suit them.
  • I3 (world): long tales could be processed in overlapping chunks with entity and state findings merged per document, while keeping chunk offsets so each step stays explainable.
  • I4 (learn): when adding a new treebank or silver batch, keep the parser fixed and compare against the previous run as a control to attribute gains honestly.
  • I5 (learn): silver sentences that are very long or carry several conflicting witness annotations could be filtered out rather than forced into a coarse label.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Rosenheim Technical University of Applied Sciences.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksClinical Psychology · 21 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksInformation extraction · 175 works

Cite

Cite as Felix Haas and Sebastian P. Bayerl. 2025. Multilingual Stutter Event Detection for English, German, and Mandarin Speech. arXiv:2603.26939 · Text, Speech, and Dialogue. TSD 2025. Lecture Notes in Computer Science(), vol 16,029.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/978-3-032-02548-7_17 · arXiv 2603.26939

@misc{haas2025multilingual,
  title = {{Multilingual Stutter Event Detection for English, German, and Mandarin Speech}},
  author = {Felix Haas and Sebastian P. Bayerl},
  year = {2025},
  doi = {10.1007/978-3-032-02548-7\_17},
  eprint = {2603.26939},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/978-3-032-02548-7_17}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 12449–12460. Curran Associates, Inc. (2020)
  2. Batra, A., Narang, M., Sharma, N.K., Das, P.K.: Boli: A dataset for understanding stuttering experience and analyzing stuttered speech. In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–4. IEEE, Hyderabad, India (Apr 2025) find
  3. Bayerl, S., Wolff von Gudenberg, A., Hönig, F., Noeth, E., Riedhammer, K.: KSoF: The Kassel State of Fluency Dataset – A Therapy Centered Dataset of Stuttering. In: Proceedings of the Language Resources and Evaluation Conference. pp. 1780– 1787. European Language Resources Association, Marseille, France (Jun 2022) find
  4. Bayerl, S.P., Wagner, D., Nöth, E., Bocklet, T., Riedhammer, K.: The Influence of Dataset Partitioning on Dysfluency Detection Systems. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) Text, Speech, and Dialogue. vol. 13502, pp. 423–436. Springer International Publishing, Cham (2022) find
  5. Bayerl, S.P., Wagner, D., Baumann, I., Hönig, F., Bocklet, T., Nöth, E., Riedhammer, K.: A Stutter Seldom Comes Alone – Cross-Corpus Stuttering Detection as a Multi-Label Problem. In: Proc. INTERSPEECH 2023. pp. 1538–1542. Dublin (2023) find
  6. Bayerl, S.P., Wagner, D., Noeth, E., Riedhammer, K.: Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0. In: Proc. Interspeech 2022. pp. 2868–2872. ISCA, Incheon (Sep 2022) find
  7. Bernstein Ratner, N., MacWhinney, B.: Fluency Bank: A new resource for fluency research and practice. Journal of Fluency Disorders 56, 69–80 (Jun 2018) find
  8. Bloodstein, O., Ratner, N.B., Brundage, S.B.: A Handbook of Stuttering. Plural Publishing, Inc, San Diego, CA, seventh edition edn. (2021) find
  9. Cai, X., Yuan, J., Zheng, R., Huang, L., Church, K.: Speech Emotion Recognition with Multi-Task Learning. In: Interspeech 2021. pp. 4508–4512. ISCA (Aug 2021) find
  10. Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., Wei, F.: WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing 16(6) (2022) find
  11. Chiu, C.C., Qin, J., Zhang, Y., Yu, J., Wu, Y.: Self-supervised learning with random-projection quantizer for speech recognition. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 3915–3924. PMLR (2022-07-17/2022-07-23) Multilingual Stutter Event Detection F. Haas find
  12. Duyen, T.M.T.: Exploring Phonetic Differences and Cross-Linguistic Influences: A Comparative Study of English and Mandarin Chinese Pronunciation Patterns. Open Journal of Applied Sciences 14(07), 1807–1822 (2024) find
  13. Ellis, J.B., Ramig, P.R.: A Handbook on Stuttering. Journal of Fluency Disorders 34(4), 295–299 (Dec 2009) find
  14. Gong, R., Xue, H., Wang, L., Xu, X., Li, Q., Xie, L., Bu, H., Wu, S., Zhou, J., Qin, Y., Zhang, B., Du, J., Bin, J., Li, M.: AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection. In: Interspeech 2024. pp. 5098–5102. ISCA (Sep 2024) find
  15. Howell, P., Davis, S., Bartrip, J.: The University College London Archive of Stuttered Speech (UCLASS). Journal of Speech, Language, and Hearing Research 52(2), 556–569 (Apr 2009) find
  16. Howell, P., Sackin, S.: Automatic recognition of repetitions and prolongations in stuttered speech. In: Proceedings of the First World Congress on Fluency Disorders. vol. 2, pp. 372–374. University Press Nijmegen Nijmegen, The Netherlands (1995) find
  17. Kourkounakis, T.: LibriStutter (2021) find
  18. Kourkounakis, T., Hajavi, A., Etemad, A.: Detecting Multiple Speech Disfluencies using a Deep Residual Network with Bidirectional Long Short-Term Memory. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6089–6093. IEEE (2020) find
  19. Lea, C., Mitra, V., Joshi, A., Kajarekar, S., Bigham, J.P.: SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6798–6802. IEEE, Toronto, ON, Canada (Jun 2021) find
  20. Lickley, R.: Disfluency in typical and stuttered speech. Fattori sociali e biologici nella variazione fonetica (3), 373 (2017) find
  21. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318–327 (Feb 2020) find
  22. Noeth, E., Niemann, H., Haderlein, T., Decher, M., Eysholdt, U., Rosanowski, F., Wittenberg, T.: Automatic stuttering recognition using hidden Markov models. In: Sixth International Conference on Spoken Language Processing. vol. 4, pp. 65–68 (Jan 2000) find
  23. Pironkov, G., Dupont, S., Dutoit, T.: Speaker-aware long short-term memory multi-task learning for speech recognition. In: 2016 24th European Signal Processing Conference (EUSIPCO). pp. 1911–1915. IEEE, Budapest, Hungary find
  24. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust Speech Recognition via Large-Scale Weak Supervision p. 28 (2022)
  25. Ravanelli, M., Zhong, J., Pascual, S., Swietojanski, P., Monteiro, J., Trmal, J., Bengio, Y.: Multi-Task Self-Supervised Learning for Robust Speech Recognition. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6989–6993. IEEE, Barcelona, Spain (May 2020) find
  26. Sheikh, S.A., Hirsch, F., Ouni, S.: Robust Stuttering Detection via Multi-task and Adversarial Learning. In: 2022 30th European Signal Processing Conference (EUSIPCO). p. 5 (2022) find
  27. Sheikh, S.A., Sahidullah, M., Hirsch, F., Ouni, S.: StutterNet: Stuttering Detection Using Time Delay Neural Network. In: 2021 29th European Signal Processing Conference (EUSIPCO). pp. 426–430. IEEE, Dublin, Ireland (Aug 2021) find
  28. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-Art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics (Oct 2020)
  29. Xu, X., Kang, Y., Cao, S., Lin, B., Ma, L.: Explore wav2vec 2.0 for Mispronunciation Detection. In: Interspeech 2021. pp. 4428–4432. ISCA (Aug 2021) find
  30. Xue, H., Gong, R., Shao, M., Xu, X., Wang, L., Xie, L., Bu, H., Zhou, J., Qin, Y., Du, J., Li, M., Zhang, B., Jia, B.: Findings of the 2024 Mandarin Stuttering Event Detection and Automatic Speech Recognition Challenge. In: 2024 IEEE Spoken Language Technology Workshop (SLT). pp. 385–392. IEEE, Macao (Dec 2024) find
  31. Yairi, E., Ambrose, N.: Epidemiology of stuttering: 21st century advances. Journal of Fluency Disorders 38(2), 66–87 (Jun 2013) find
  32. Zhou, X., Kashyap, A., Li, S., Sharma, A., Morin, B., Baquirin, D., Vonk, J., Ezzes, Z., Miller, Z., Tempini, M., Lian, J., Anumanchipalli, G.: YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection. In: Interspeech 2024. pp. 937–941. ISCA (Sep 2024) Multilingual Stutter Event Detection find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haas-2026-multilingual-stutter-event-detection-english/