Elamur Research⌕
☰
home / papers / 2026 / Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study

Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study

Paper · 2026 · Yuanyuan Zhang, Dimme de Groot, Jorge Martinez et al.

Summary

The study asks how well people and machines understand one Dutch speaker with severe dysarthria. It also asks whether models adapted to that speaker can beat untrained human listeners. The material comes from DysOne, a dataset under development that records a 35-year-old man whose dysarthria followed an acquired brain injury. The 188 test utterances mix two kinds of read prompts, standard ones and ones tailored to the speaker, with spontaneous speech, which no earlier comparison of this kind had included. Twenty native Dutch listeners with no prior exposure to dysarthric speech typed what they heard. Three off-the-shelf systems transcribed the same clips: Whisper-large-V3, Google Chirp 3 and Omnilingual ASR. Whisper and Omnilingual were also fine-tuned on the rest of the speaker's recordings. Both the listeners and the untuned systems averaged word error rates above 70%. The personalised models brought errors down sharply. Their WER stayed above 23%, yet they outperformed the human listeners. This suggests speaker-specific recognition is nearing practical use in everyday conversation with unfamiliar people. Spontaneous speech, longer read utterances and certain phonemes remain the weak spots.

Key points

  • The Dutch part of DysOne holds about 3.3 h of read and 0.4 h of spontaneous speech. Generic read prompts come from the Corpus Gesproken Nederlands, in length bins of (1,5], (5,10] and (10,15] words.
  • The test set has 188 utterances with 686 unique words: 121 generic read, 27 customized read (statistics-lecture content chosen by the speaker) and 40 spontaneous. Training and validation use 1304 and 165 utterances.
  • The 20 listeners were split across four lists of 47 items each, with every list heard by five people. Replays were unlimited but logged, and no feedback was given during the task.
  • The systems tested were Whisper-large-V3 (Dutch, transcribe mode, temperature 0), Google Chirp 3 via its synchronous API, and omniASR-LLM-300M-v2. The fine-tuning used speed perturbation at 0.9 and 1.1, plus LoRA for Whisper.
  • Humans and zero-shot ASR both averaged WER above 70%. The personalised models stayed above 23% but did better than the listeners.

Interesting ideas

  • I1. People and every system were scored on exactly the same clips, so the human baseline is a direct reference point rather than a figure borrowed from another study.
  • I2. Results are broken down by speech type and utterance-length bin, which shows where failures concentrate (spontaneous speech, long utterances) instead of hiding them in one average.
  • I3. A strong general model is adapted to one narrow target with very little data, using a lightweight method (LoRA) and cheap augmentation (speed perturbation).
  • I4. Listeners got no corrections during the task, to avoid learning effects that would distort the scores. Their replay counts were logged as an extra signal of item difficulty.
  • I5. One test subset was built from content the end user actually cares about (the speaker's own lecture topics), alongside generic material.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Have human annotators and the witness parsers label one shared held-out sentence set, so parser LAS can be read against a measured human ceiling.
  • I2 (tree): Report LAS for perceptron and gparse by sentence-length bucket and genre (read-like prose versus conversational text) to target repair rules and training data.
  • I3 (learn): Domain adapters on gparse, trained on small silver sets for a specific text type such as fables or maths problems, could lift accuracy without retraining the whole parser.
  • I4 (learn): When an LLM teacher or self-play loop is evaluated, keep gold labels away from the system under test, and log retry or backtracking counts per item as a difficulty measure.
  • I5 (answer): Pair standard test sets with a small user-domain set, such as a specific school course, to check that reader performance carries over to the texts people will really ask about.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Delft University of TechnologyDelft · Netherlands6 works · 290 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 worksMedicine · 275 worksPhysiology · 35 works

Cite

Cite as Yuanyuan Zhang, Dimme de Groot, Jorge Martinez and Odette Scharenborg. 2026. Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study. arXiv:2606.30237.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.30237 · arXiv 2606.30237

@misc{zhang2026comparing,
  title = {{Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study}},
  author = {Yuanyuan Zhang, Dimme {de Groot}, Jorge Martinez and Odette Scharenborg},
  year = {2026},
  doi = {10.48550/arxiv.2606.30237},
  eprint = {2606.30237},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.30237}
}

References

60 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Cooke and O. Scharenborg, “The interspeech 2008 consonant challenge,” in Interspeech, 2008, pp. 1765–1768. find
  2. R. P. Lippmann, “Speech recognition by machines and humans,” Speech communication, vol. 22, no. 1, pp. 1–15, 1997. ↗
  3. C. Spille, B. Kollmeier, and B. T. Meyer, “Comparing human and automatic speech recognition in simple and complex acoustic scenes,” Computer Speech & Language, vol. 52, pp. 123–140, 2018. ↗
  4. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Achieving human parity in conversational speech recognition,” arXiv preprint arXiv:1610.05256, 2016. ↗
  5. C. Patman and E. Chodroff, “Speech recognition in adverse conditions by humans and machines.” Journal of the Acoustical Society of America - Express Letters, vol. 4 11, 2024. ↗
  6. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  7. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449– 12 460, 2020.
  8. Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu, “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037, 2023. ↗
  9. A. Davitaia, “Applications of machine learning in speech recognition,” Available at SSRN 5329566, 2025. ↗
  10. C. Wienrich, C. Reitelbach, and A. Carolus, “The trustworthiness of voice assistants in the context of healthcare investigating the effect of perceived expertise on the trustworthiness of voice assistants, providers, data receivers, and automatic speech recognition,” Frontiers in Computer Science, vol. 3, p. 685250, 2021. ↗
  11. Y. Zhang, L. Xu, A. Mendoza, G. Yang, P. Chinprutthiwong, and G. Gu, “Life after speech recognition: Fuzzing semantic misinterpretation for voice assistant applications,” in Proceedings of the of the Network and Distributed System Security Symposium (NDSS’19), 2019. find
  12. X. Luo, L. Zhou, K. Adelgais, and Z. Zhang, “Assessing the effectiveness of automatic speech recognition technology in emergency medicine settings: a comparative study of four ai-powered engines,” Journal of Healthcare Informatics Research, pp. 1–19, 2025. ↗
  13. A. Elhadad, I. Alrashdi, A. M. Albarrak, S. R. I. Elrefaey, H. A. E. Elsayed, F. M. Embarak, Z. Ulmas, and Y. A. B. El-Ebiary, “Improved healthcare diagnosis accuracy through the application of deep learning techniques in medical transcription for disease identification,” Alexandria Engineering Journal, vol. 123, pp. 112–123, 2025. ↗
  14. Y. Kumar, “A comprehensive analysis of speech recognition systems in healthcare: current research challenges and future prospects,” SN Computer Science, vol. 5, no. 1, p. 137, 2024. ↗
  15. Y. Zhang, A. Herygers, T. Patel, Z. Yue, and O. Scharenborg, “Exploring data augmentation in bias mitigation against non-native-accented speech,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8. find
  16. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518. ↗
  17. C. Patman and E. Chodroff, “Speech recognition in adverse conditions by humans and machines,” JASA Express Letters, vol. 4, no. 11, 2024. ↗
  18. Z. Tüske, G. Saon, and B. Kingsbury, “On the Limit of English Conversational Speech Recognition,” in Interspeech 2021, 2021, pp. 2062–2066.
  19. A. Faria, A. Janin, S. Adkoli, and K. Riedhammer, “Toward Zero Oracle Word Error Rate on the Switchboard Benchmark,” in Interspeech 2022, 2022, pp. 3973–3977. find
  20. Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang et al., “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1519–1532, 2022. ↗
  21. W. Xiong, J. Droppo, X. Huang, F. Seide, M. L. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Toward human parity in conversational speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2410–2423, 2017. ↗
  22. G. Saon, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, G. Kurata, L. Lim et al., “English conversational telephone speech recognition by humans and machines,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2017, pp. 132–136.
  23. L. De Russis and F. Corno, “On the impact of dysarthric speech on contemporary asr cloud platforms,” Journal of Reliable Intelligent Environments, vol. 5, no. 3, pp. 163–172, 2019. ↗
  24. F. Rudzicz, A. K. Namasivayam, and T. Wolff, “The torgo database of acoustic and articulatory speech from speakers with dysarthria,” Language resources and evaluation, vol. 46, no. 4, pp. 523–541, 2012. ↗
  25. A. Alsayegh and T. Masood, “Zero-shot recognition of dysarthric speech using commercial automatic speech recognition and multimodal large language models,” arXiv preprint arXiv:2512.17474, 2025. ↗
  26. T. Bent, M. Baese-Berk, S. A. Borrie, and M. McKee, “Individual differences in the perception of regional, nonnative, and disordered speech varieties,” The Journal of the Acoustical Society of America, vol. 140, no. 5, pp. 3775–3786, 2016. ↗
  27. K. K. Tjaden and J. M. Liss, “The role of listener familiarity in the perception of dysarthric speech,” Clinical linguistics & phonetics, vol. 9, no. 2, pp. 139–154, 1995. ↗
  28. K. T. Mengistu and F. Rudzicz, “Comparing humans and automatic speech recognition systems in recognizing dysarthric speech,” in Canadian Conference on Artificial Intelligence. Springer, 2011, pp. 291–300. find
  29. H. Kim, “Familiarization effects on consonant intelligibility in dysarthric speech,” Folia Phoniatrica et Logopaedica, vol. 67, no. 5, pp. 245–252, 2016. ↗
  30. M. Sun, M. Gao, X. Kang, S. Wang, J. Du, D. Yao, and S.-J. Wang, “Cdsd: Chinese dysarthria speech database,” arXiv preprint arXiv:2310.15930, 2023. ↗
  31. J. R. Green, R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nelson, and K. Tomanek, “Automatic Speech Recognition of Disordered Speech: Personalized Models Outperforming Human Listeners on Short Phrases,” in Interspeech 2021, 2021, pp. 4778–4782. find
  32. C. Fougeron, L. Crevier-Buchman, C. Fredouille, A. Ghio, C. Meunier, C. Chevrie-Muller, N. Audibert, J.-F. Bonastre, A. Colazo-Simon, C. Delooze et al., “Developing an acoustic-phonetic characterization of dysarthric speech in french,” in 7th International Conference on Language Resources, Technologies and Evaluation (LREC), vol. 1, no. 1. Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard . . . , 2010, pp. 2831–2838. find
  33. H. Christensen, S. Cunningham, C. Fox, P. Green, and T. Hain, “A comparative study of adaptive, automatic recognition of disordered speech,” in Interspeech 2012, 2012, pp. 1776–1779. find
  34. R. Vinotha, D. Hepsiba, L. Vijay Anand, J. Andrew, and R. Jennifer Eunice, “Enhancing dysarthric speech recognition through sepformer and hierarchical attention network models with multistage transfer learning,” Scientific Reports, vol. 14, no. 1, p. 29455, 2024. ↗
  35. E. Sanders, M. Ruiter, L. Beijer, and H. Strik, “Automatic recognition of Dutch dysarthric speech: A pilot study,” in Interspeech. ISCA, Sep. 2002, pp. 661–664. find
  36. J. Shor, D. Emanuel, O. Lang, O. Tuval, M. Brenner, J. Cattiau, F. Vieira, M. McNally, T. Charbonneau, M. Nollstadt, A. Hassidim, and Y. Matias, “Personalizing asr for dysarthric and accented speech with limited data,” in Interspeech 2019, vol. 2019-September, 2019, pp. 784–788. find
  37. J. Tobin and K. Tomanek, “Personalized automatic speech recognition trained on small disordered speech datasets,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6637–6641. ↗
  38. M. S. Hawley, P. Enderby, P. Green, S. Cunningham, S. Brownsell, J. Carmichael, M. Parker, A. Hatzis, P. O’Neill, and R. Palmer, “A speech-controlled environmental control system for people with severe dysarthria,” Medical Engineering & Physics, vol. 29, no. 5, pp. 586–593, 2007. ↗
  39. O. Scharenborg, “Inclusive speech technology: Developing automatic speech recognition for everyone,” Webinar delivered to the TU Delft Safety and Security Institute and Campus, The Hague, the Netherlands, 2021. find
  40. OpenAI, “Whisper-large-v3,” 2023. [Online]. Available: https://huggingface.co/openai/whisper-large-v3 ↗
  41. Google Cloud, “Speech-to-text: Transcription models,” https://cloud.google.com/speech-to-text/docs/transcription-model, 2026, accessed: 2026-02-18. ↗
  42. A. Omnilingual, G. Keren, A. Kozhevnikov, Y. Meng, C. Ropers, M. Setzler, S. Wang, I. Adebara, M. Auli, C. Balioglu et al., “Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages,” arXiv preprint arXiv:2511.09690, 2025. ↗
  43. N. Oostdijk, “The Spoken Dutch Corpus. Overview and first evaluation,” in Proc. of the 2nd International Conference on Language Resources and Evaluation. Athens, Greece: European Language Resources Association, May 2000. find
  44. “Streamlit,” 2026. [Online]. Available: https://streamlit.io ↗
  45. M. H. Davis, I. S. Johnsrude, A. Hervais-Adelman, K. Taylor, and C. McGettigan, “Lexical information drives perceptual learning of distorted speech: evidence from the comprehension of noise-vocoded sentences.” Journal of Experimental Psychology: General, vol. 134, no. 2, p. 222, 2005. ↗
  46. Microsoft, “Language and voice support for the speech service,” https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt, 2025, accessed: 2026-06-21. ↗
  47. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024. ↗
  48. Y. Zhang, T. De Valck, and O. Scharenborg, “Speech recognition performance disparities between dutch diverse speaker groups,” Phonetica, no. 0, 2026. ↗
  49. N. R. Koluguri, M. Sekoyan, G. Zelenfroynd, S. Meister, S. Ding, S. Kostandian, H. Huang, N. Karpov, J. Balam, V. Lavrukhin et al., “Granary: Speech recognition and translation dataset in 25 european languages,” arXiv preprint arXiv:2505.13404, 2025. ↗
  50. The Omnilingual ASR Team, “omniasr-llm-300m-v2,” 2025. [Online]. Available: https://huggingface.co/bezzam/omniasr-llm-300m-v2 ↗
  51. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022.
  52. M. Bisani and H. Ney, “Bootstrap estimates for confidence intervals in asr performance evaluation,” in IEEE ICASSP, vol. 1, 2004, pp. I–409. find
  53. L. Ferrer, O. Scharenborg, and T. Bäckström, “Good practices for evaluation of machine learning systems,” arXiv preprint arXiv:2412.03700, 2024. ↗
  54. D. D. Boos and L. A. Stefanski, Essential Statistical Inference: Theory and Methods. Springer, 2013, section 11.6: Bootstrap Resampling for Hypothesis Tests. find
  55. R. Lenth, “Emmeans: Estimated marginal means, aka least-squares means .” R package version 2.0. 1, 2023. find
  56. S. Feng, B. M. Halpern, O. Kudina, and O. Scharenborg, “Towards inclusive automatic speech recognition,” Computer Speech & Language, vol. 84, p. 101567, 2024. ↗
  57. M. Fuckner, S. Horsman, P. Wiggers, and I. Janssen, “Uncovering bias in asr systems: Evaluating wav2vec2 and whisper for dutch speakers,” in 2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD). IEEE, 2023, pp. 146–151. find
  58. B. M. Halpern, S. Feng, R. van Son, M. van den Brekel, and O. Scharenborg, “Low-resource automatic speech recognition and error analyses of oral cancer speech,” Speech Communication, vol. 141, pp. 14–27, 2022. ↗
  59. B. M. Halpern, “Relative phoneme analysis,” 2020. [Online]. Available: https://github.com/karkirowle/relative phoneme analysis ↗
  60. G. Hilkhuysen, J. Lloyd, and M. Huckvale, “Effects of replay on the iin-telligibility of noisy speech,” in Audio Engineering Society Conference: 46th International Conference: Audio Forensics. Audio Engineering Society, 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-comparing-human-automatic-recognition-dutch/