Elamur Research⌕
☰
home / papers / 2021 / Arabic Speech Recognition by End-to-End, Modular Systems and Human

Arabic Speech Recognition by End-to-End, Modular Systems and Human

Summary

The paper asks whether machines now match people at transcribing Arabic, a morphologically rich language with mostly undiacritized text and many dialects that lack standard spelling. Earlier human-versus-machine comparisons covered only English and modular HMM-DNN systems. The authors built what they describe as the first end-to-end transformer ASR for Arabic in ESPnet. It uses sub-word units to limit out-of-vocabulary words. They compared it head to head with a strong single HMM-DNN system and with human transcribers, both expert linguists and lay native speakers. For this comparison they recorded a new hidden 3-hour test set mixing news and conversation in MSA and dialects. The transformer set new best results on MGB2, MGB3 and MGB5. It arguably beats lay natives, but expert linguists still lead by about 3.5% absolute WER on average. The authors also found that long, unsegmented audio badly hurts the transformer, and they designed a VAD pipeline that caps segment length. The work matters because it gives a clear benchmark and public recipe for Arabic and its dialects.

Key points

  • The E2E transformer reached 12.5% WER on MGB2, 27.5% on MGB3 and 33.8% on MGB5. The earlier best on MGB2 was Aalto's 13.2%, which combined more than 30 systems.
  • On dialect tests, AV-WER fell from 37.5 to 34.4 on MGB3 and from 59.4 (RDI-CU) to 56.2 on MGB5 when adaptation and dev data were both used for finetuning.
  • In the human study, machine errors closely resembled those of expert linguists. The remaining gap to experts averages 3.5% absolute WER on raw transcripts.
  • The VAD pipeline merges InaSpeechSegmenter's speech detection with the length cap of an energy-based VAD.
    • Plain InaSeg left 43.6% of segments longer than 30 s, against 7.9% in training data, and the model broke down beyond roughly 30 s.
    • The improved segmentation keeps every segment under 30 s.
  • For the dialect sets, the MGB2 model was finetuned with a lower learning rate (0.1) and no warm-up. Transcripts from four annotators per utterance were pooled, giving four times the training text.

Interesting ideas

  • I1. The authors built a fresh, unpublished test set because public benchmarks may already have shaped earlier systems, which keeps the comparison honest. — For Mova (learn): Keep a private held-out set of texts and questions that no teacher tier or tuning loop ever sees, and use it as the final check alongside negative controls.
  • I2. They used two levels of human baseline (expert and lay native) and analysed how machine errors correlate with each, which shows the kind of errors made, not just their number. — For Mova (answer): Benchmark the readers on school tests against both strong and average human answers, and compare error overlap to see whether failures look like plausible human mistakes or like system bugs.
  • I3. They explained a sudden accuracy collapse by comparing input length distributions in training and test data, then fixed it by capping length at the source. — For Mova (tree): Track sentence-length distributions for gparse training treebanks against incoming text, and split or flag overlong inputs before parsing instead of trusting the model past its seen range.
  • I4. When several annotators gave different but valid transcripts, each one was kept as a separate training target and scored with a multi-reference metric. — For Mova (learn): When witness parsers or annotators disagree on equally valid analyses, keep the alternatives as multiple acceptable references instead of forcing one silver answer.
  • I5. They pretrained on the standard variety and then finetuned gently, with a lower rate and no warm-up, on small dialect sets, a simple recipe for low-resource varieties. — For Mova (tree): Train on large standard-English UD treebanks first, then lightly adapt to small treebanks of other registers or UD dialects after conversion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2101.08454 · arXiv 2101.08454

@misc{hussein2021arabic,
  title = {{Arabic Speech Recognition by End-to-End, Modular Systems and Human}},
  author = {Hussein, Amir and Watanabe, Shinji and Ali, Ahmed},
  year = {2021},
  doi = {10.48550/arxiv.2101.08454},
  eprint = {2101.08454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2101.08454}
}

References

43 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al., Deep speech 2: End-to-end speech recognition in English and Mandarin, in: International conference on machine learning, 2016, pp. 173–182. find our card
  2. A. Graves, N. Jaitly, Towards end-to-end speech recognition with recurrent neural networks, in: International conference on machine learning, 2014, pp. 1764–1772. find
  3. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig, Achieving human parity in conversational speech recognition, arXiv preprint arXiv:1610.05256. ↗
  4. G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, et al., English conversational telephone speech recognition by humans and machines, Proc. Interspeech (2017) 132–136. find
  5. P. Michalowski, The lives of the sumerian language, Margins of writing, origins of cultures (2006) 159–84. find
  6. P. Smit, S. R. Gangireddy, S. Enarvi, S. Virpioja, M. Kurimo, Aalto system for the 2017 Arabic multi-genre broadcast challenge, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2017, pp. 338–345. find
  7. H. Mubarak, A. Abdelali, H. Sajjad, Y. Samih, K. Darwish, Highly effective arabic diacritization using sequence to sequence modeling, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 2390–2395. find
  8. A. Das, M. Hasegawa-Johnson, Cross-lingual transfer learning during supervised training in low resource scenarios, in: Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  9. S. Khurana, A. Ali, J. Glass, Darts: Dialectal Arabic transcription system, arXiv preprint arXiv:1909.12163. ↗ our card
  10. A. Ahmed, Y. Hifny, K. Shaalan, S. Toral, End-to-end lexicon free Arabic speech recognition using recurrent neural networks, Computational Linguistics, Speech And Image Processing For Arabic Language 4 (2018) 231. find
  11. T. Kudo, Subword regularization: Improving neural network translation models with multiple subword candidates, arXiv preprint arXiv:1804.10959. ↗
  12. S. Khurana, A. Ali, Qcri advanced transcription system (QATS) for the Arabic multi-dialect broadcast media recognition: MGB-2 challenge, in: IEEE Spoken Language Technology Workshop (SLT), IEEE, 2016, pp. 292– 298. find
  13. D. Doukhan, E. Lechapt, M. Evrard, J. Carrive, Ina’s mirex 2018 music and speech detection system, in: Music Information Retrieval Evaluation eXchange (MIREX 2018), 2018. find
  14. T. Giannakopoulos, pyaudioanalysis: An open-source python library for audio signal analysis, PloS one 10 (12) (2015) e0144610. find
  15. G. E. Dahl, D. Yu, L. Deng, A. Acero, Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, IEEE Transactions on audio, speech, and language processing 20 (1) (2011) 30–42. find
  16. A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in: IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, 2013, pp. 6645–6649. find
  17. Y. Hifny, Unified acoustic modeling using deep conditional random fields, Transactions on Machine Learning and Artificial Intelligence 3 (2) (2015) 65–65. find
  18. V. Peddinti, D. Povey, S. Khudanpur, A time delay neural network architecture for efficient modeling of long temporal contexts, in: Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  19. A. Ali, S. Vogel, S. Renals, Speech recognition challenge in the wild: Arabic MGB-3, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2017, pp. 316–322. find our card
  20. D. Wang, X. Wang, S. Lv, An overview of end-to-end automatic speech recognition, Symmetry 11 (8) (2019) 1018. find
  21. W. Chan, N. Jaitly, Q. Le, O. Vinyals, Listen, attend and spell: A neural network for large vocabulary conversational speech recognition, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2016, pp. 4960–4964. find
  22. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, Y. Bengio, Attention-based models for speech recognition, Advances in neural information processing systems 28 (2015) 577–585. find our card
  23. S. Watanabe, T. Hori, S. Kim, J. R. Hershey, T. Hayashi, Hybrid ctc/attention architecture for end-to-end speech recognition, IEEE Journal of Selected Topics in Signal Processing 11 (8) (2017) 1240–1253. find
  24. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. find
  25. S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang, et al., A comparative study on transformer vs rnn in speech applications, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019, pp. 449– 456. find
  26. Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang, et al., Transformer-based acoustic modeling for hybrid speech recognition, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 6874– 6878. find
  27. G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, R. Collobert, End-to-end ASR: from supervised to semi-supervised learning with modern architectures, arXiv preprint arXiv:1911.08460. ↗
  28. A. Stolcke, J. Droppo, Comparing human and machine errors in conversational speech transcription, arXiv preprint arXiv:1708.08615. ↗
  29. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, Y. Zhang, The mgb-2 challenge: Arabic multi-dialect broadcast media recognition, in: IEEE Spoken Language Technology Workshop (SLT), IEEE, 2016, pp. 279–284. find our card
  30. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, S. Khudanpur, Purely sequence-trained neural networks for ASR based on lattice-free MMI, in: Proc. Interspeech, 2016, pp. 2751–2755. find
  31. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al., The kaldi speech recognition toolkit, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), no. CONF, IEEE Signal Processing Society, 2011. find
  32. D. Povey, H. Hadian, P. Ghahremani, K. Li, S. Khudanpur, A time-restricted self-attention layer for ASR, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2018, pp. 5874–5878. find
  33. P. Ghahremani, B. BabaAli, D. Povey, K. Riedhammer, J. Trmal, S. Khudanpur, A pitch extraction algorithm tuned for automatic speech recognition, in: IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, 2014, pp. 2494–2498. find
  34. J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450. ↗ our card
  35. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. find
  36. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, K. Choukri, The MGB-5 challenge: Recognition and dialect identification of dialectal Arabic speech, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019, pp. 1026–1033. find
  37. A. Ali, W. Magdy, P. Bell, S. Renais, Multi-reference wer for evaluating asr for languages with no orthographic rules, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2015, pp. 576– 580. find
  38. S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N.-E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, et al., Espnet: End-to-end speech processing toolkit, Proc. Interspeech (2018) 2207–2211. find
  39. A. Ali, Y. Zhang, P. Cardinal, N. Dahak, S. Vogel, J. Glass, A complete kaldi recipe for building arabic speech recognition systems, in: IEEE spoken language technology workshop (SLT), IEEE, 2014, pp. 525–529. find
  40. T. Ko, V. Peddinti, D. Povey, S. Khudanpur, Audio augmentation for speech recognition, in: Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  41. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, Q. V. Le, Specaugment: A simple data augmentation method for automatic speech recognition, Proc. Interspeech (2019) 2613–2617. find our card
  42. R. Pappagari, P. Zelasko, J. Villalba, Y. Carmiel, N. Dehak, Hierarchical transformers for long document classification, in: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019, pp. 838– 844. find
  43. A. Zeyer, K. Irie, R. Schlüter, H. Ney, Improved training of end-to-end attention models for speech recognition, arXiv preprint arXiv:1805.03294. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hussein-2021-arabic-speech-recognition-end-end/