Elamur Research⌕
☰
home / papers / 2023 / Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation

Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation

Summary

Speech recognizers that work well for typical speakers fail on dysarthric speech, a motor disorder that makes articulation slow, slurred or prosodically irregular. In Arabic no dysarthric corpus exists, and recording patients is difficult because of fatigue and low compliance. The authors therefore synthesize training data. First, healthy Arabic recordings from Common Voice are altered with speed perturbation (resampling) and tempo perturbation (WSOLA). Second, a Parallel WaveGAN (PWG) trained on the English dysarthric Torgo corpus re-synthesizes these signals. This rests on the assumption that dysarthric traits are largely language-independent. The combined synthetic audio is used to fine-tune a Conformer previously trained on MGB-2. A post-processing step then replaces output words with Arabic Wikipedia lexicon entries, using a Jaccard distance weighted by character confusions. On synthetic dysarthric Common Voice speech the system reaches 18% WER and 17.2% CER, well ahead of a healthy-only baseline. A parallel English check on real dysarthric speech also shows large gains. The work offers a recipe for disordered-speech ASR when no in-language patient data exists.

Key points

  • Results: 18% WER and 17.2% CER on synthetically "dysarthrified" Arabic Common Voice, reported as an 81.8% WER improvement over a Conformer trained only on healthy speech.
  • English validation: a baseline trained on healthy LJSpeech is compared with the augmented approach on real dysarthric speech, with a reported 124% WER improvement.
  • Data: an 8-hour gender-balanced Common Voice Arabic subset (resampled to 16 kHz) is the augmentation source. A 20-hour MGB-2 subset (18 h train, 1 h dev, 1 h test) trains the base Arabic ASR. Torgo supplies English dysarthric speech, and a 4-hour LJSpeech subset is expanded into 16 hours of synthetic dysarthric English.
  • Models: ESPnet Conformer (Conformer encoder plus Transformer decoder) for ASR. Parallel WaveGAN, with a WaveNet-style generator conditioned on mel-spectrograms plus noise, for adversarial augmentation.
  • Text correction uses a weighted Jaccard distance in which frequently confused characters count as closer. Each predicted word is mapped to its nearest entry in a dictionary built from an Arabic Wikipedia dump.

Interesting ideas

  • I1. Rare conditions can be simulated by applying simple, controllable distortions (here, speed and tempo) to plentiful clean data, giving a synthetic training set when real examples cannot be collected. — For Mova (tokens): Systematically corrupting clean treebank text with typos, dropped apostrophes or casing noise could give the tagger and lemmatizer training material for messy input, with each corruption type labeled and traceable.
  • I2. The augmentation runs in two stages: a cheap rule-based transformation first, then a learned model that adds subtler properties the rules cannot express. — For Mova (learn): Silver data could be built the same way, with deterministic rewrite rules producing candidates and a learned or LLM teacher stage refining them, while keeping the rule layer inspectable.
  • I3. The model's own observed confusions weight the edit distance used to snap noisy outputs onto a closed lexicon, so correction follows the errors the system actually makes. — For Mova (tokens): A lemma or spelling normalizer could weight edits by the confusions logged in Mova's own tagging errors, and each correction would carry an explanation of the swap.
  • I4. Patterns learned in a resource-rich language are transferred to a target language, on the assumption that the phenomenon is largely language-independent. — For Mova (frames): Role or speech-act cues annotated richly in one resource could be tested for transfer to another genre or dialect, provided the transfer assumption is checked explicitly.
  • I5. The synthetic-data pipeline is validated in a language that has real impaired speech, as a sanity check before trusting purely synthetic evaluation in the target language. — For Mova (learn): Silver-data and repair-rule pipelines should be scored on a domain with gold UD annotation before results on silver-only domains are believed, much like a negative or positive control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech & spoken language · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.04368 · arXiv 2306.04368

@misc{baali2023arabic,
  title = {{Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation}},
  author = {Baali, Massa and Almakky, Ibrahim and Shehata, Shady and Karray, Fakhri},
  year = {2023},
  doi = {10.48550/arxiv.2306.04368},
  eprint = {2306.04368},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.04368}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al., “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International conference on machine learning, 2016. find our card
  2. R. Turrisi, A. Braccia, M. Emanuele, S. Giulietti, M. Pugliatti, M. Sensi, L. Fadiga, and L. Badino, “Easycall corpus: a dysarthric speech dataset,” arXiv, 2021. find
  3. S. Hu, X. Xie, M. Geng, M. Cui, J. Deng, T. Wang, X. Liu, and H. Meng, “Exploiting cross-domain and cross-lingual ultrasound tongue imaging features for elderly and dysarthric speech recognition,” arXiv preprint arXiv:2206.07327, 2022. ↗
  4. Z. Yue, E. Loweimi, and Z. Cvetkovic, “Raw source and filter modelling for dysarthric speech recognition,” in ICASSP. IEEE, 2022. find
  5. M. Geng, X. Xie, Z. Ye, T. Wang, G. Li, S. Hu, X. Liu, and H. Meng, “Speaker adaptation using spectro-temporal deep features for dysarthric and elderly speech recognition,” arXiv preprint arXiv:2202.10290, 2022. ↗
  6. N. M. Joy and S. Umesh, “Improving acoustic models in torgo dysarthric speech database,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018. find
  7. X. Xie, R. Ruzi, X. Liu, and L. Wang, “Variational auto-encoder based variability encoding for dysarthric speech recognition,” in Interspeech 2021. find
  8. A. Hernandez, P. A. Pérez-Toro, E. Nöth, J. R. Orozco-Arroyave, A. Maier, and S. H. Yang, “Cross-lingual self-supervised speech representations for improved dysarthric speech recognition,” in Interspeech 2022. find
  9. C. Espana-Bonet and J. A. Fonollosa, “Automatic speech recognition with deep neural networks for impaired speech,” in International Conference on Advances in Speech and Language Technologies for Iberian Languages. Springer, 2016. find
  10. M. Soleymanpour, M. T. Johnson, R. Soleymanpour, and J. Berry, “Synthesizing dysarthric speech using multi-speaker tts for dysarthric speech recognition,” in ICASSP, 2022. find
  11. J. Ren and M. Liu, “An automatic dysarthric speech recognition approach using deep neural networks,” International Journal of Advanced Computer Science and Applications, 2017. find
  12. Y. Matsuzaka, R. Takashima, C. Sasaki, and T. Takiguchi, “Data augmentation for dysarthric speech recognition based on text-to-speech synthesis,” in 2022 IEEE 4th Global Conference on Life Sciences and Technologies. find
  13. M. Dhanalakshmi and P. Vijayalakshmi, “Intelligibility modification of dysarthric speech using hmm-based adaptive synthesis system,” in ICoBE. IEEE, 2015. find
  14. B. Vachhani, C. Bhat, and S. K. Kopparapu, “Data augmentation using healthy speech for dysarthric speech recognition.” in Interspeech, 2018. find
  15. M. Geng, X. Xie, S. Liu, J. Yu, S. Hu, X. Liu, and H. Meng, “Investigation of data augmentation techniques for disordered speech recognition,” Interspeech, pp. 696–700, 2020. find
  16. C. Bhat, A. Panda, and H. Strik, “Improved asr performance for dysarthric speech using two-stage dataaugmentation,” Interspeech 2022. find
  17. Z. Jin, M. Geng, X. Xie, J. Yu, S. Liu, X. Liu, and H. Meng, “Adversarial data augmentation for disordered speech recognition,” Proc. Interspeech 2021. find
  18. J. Yu, X. Xie, S. Liu, S. Hu, M. W. Lam, X. Wu, K. H. Wong, X. Liu, and H. Meng, “Development of the cuhk dysarthric speech recognition system for the ua speech corpus.” in Interspeech, 2018. find
  19. H. Kim, M. Hasegawa-Johnson, A. Perlman, J. Gunderson, T. S. Huang, K. Watkin, and S. Frame, “Dysarthric speech database for universal access research,” in Ninth Annual Conference of the International Speech Communication Association, 2008. find
  20. D. Wang, J. Yu, X. Wu, L. Sun, X. Liu, and H. Meng, “Improved end-to-end dysarthric speech recognition via meta-learning based model re-initialization,” in ISCSLP. IEEE, 2021. find
  21. R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in ICASSP. IEEE, 2020. find
  22. M. A. Hegazi and A. Abdelhamid, “Developing an arabic speech intelligibility test for adolescents and adults,” The Egyptian Journal of Otolaryngology, vol. 35, no. 1, pp. 86–91, 2019. find
  23. T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in Sixteenth annual conference of the international speech communication association, 2015. find
  24. W. Verhelst and M. Roelands, “An overlap-add technique based on waveform similarity (wsola) for high quality time-scale modification of speech,” in 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing. find
  25. P. Guo, F. Boyer, X. Chang, T. Hayashi, Y. Higuchi, H. Inaguma, N. Kamo, C. Li, D. Garcia-Romero, J. Shi et al., “Recent developments on espnet toolkit boosted by conformer,” in ICASSP 2021. find
  26. J. M. Hancock, “Jaccard distance (jaccard index, jaccard similarity coefficient),” Dictionary of Bioinformatics and Computational Biology, 2004. find
  27. A. Khooli, “Arabic Wiki data Dump 2018 Kaggle,” [Online; accessed 22-Oct-2022]. [Online]. Available: https://www.kaggle.com/datasets/abedkhooli/arabic-wiki-data-dump-2018 ↗
  28. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the 12th Language Resources and Evaluation Conference, 2020. find our card
  29. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in SLT 2016. find our card
  30. F. Rudzicz, A. K. Namasivayam, and T. Wolff, “The torgo database of acoustic and articulatory speech from speakers with dysarthria,” Language Resources and Evaluation, 2012. find
  31. K. Ito and L. Johnson, “The lj speech dataset,” Online: https://keithito.com/LJ-Speech-Dataset, 2017. ↗
  32. T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” EMNLP, p. 66, 2018. find our card
  33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, 2017. find our card
  34. C. Lo, S. Fu, W. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H. Wang, “Mosnet: Deep learning based objective assessment for voice conversion,” in Interspeech 2019. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baali-2023-arabic-dysarthric-speech-recognition-using/