Elamur Research⌕
☰
home / papers / 2023 / AraSpot: Arabic Spoken Command Spotting

AraSpot: Arabic Spoken Command Spotting

Summary

The paper addresses keyword spotting (KWS) for Arabic: recognising which of a small set of spoken commands occurs in a one-second clip. This task underlies voice-activated devices that have limited compute. Arabic has very little public KWS data. The authors work with the ASC dataset, which has 40 Modern Standard Arabic commands recorded by 30 speakers in quiet conditions, and attack the problem on three fronts:

  • Online augmentation. Each training step applies a random, reordered mix of time-domain and frequency-domain corruptions, such as urban noise and room reverberation.
  • ConformerGRU. A compact architecture that pairs conformer layers with a GRU, so that both local acoustic patterns and longer-range structure are captured.
  • Synthetic speech. A text-to-speech model is trained to add voices beyond the 30 real speakers.

The best configuration reaches 99.59% test accuracy, compared with the 97.97% CNN reported with the dataset. This cuts the error rate by roughly four fifths. The work shows that careful augmentation, synthetic speakers and a small hybrid model can largely compensate for scarce data in a low-resource language.

Key points

  • ASC dataset: 12,000 one-second 16 kHz mono clips covering 40 keywords. Each keyword has 300 utterances (30 speakers × 10). An extra noise/NULL class of 300 clips was added, split 60/20/20.
  • Features and training: 40 MFCCs (25 ms window, 10 ms stride, 80 filter banks), NLL loss, Adam with linearly decaying learning rate starting at 1e-3, 15% dropout, batch size 256, one RTX 3080 Ti.
  • Hyperparameter grid: d_model ∈ {64, 96, 128}, heads ∈ {2, 4}, conformer layers ∈ {1, 2}. Two layers always beat one. Extra attention heads did not help. Larger d_model did help.
  • Best model: (d_model 128, h 2, N 2), about 895K parameters. It scores 99.35% with augmentation only and 99.59% when TTS data is added.
  • Augmentation sources: noise comes from the Freesound test set, cut as a random chunk at a random offset with random gain. Reverberation comes from convolving speech with RIRs from two public datasets.

Interesting ideas

  • I1. Each augmentation is switched on independently by a per-step random draw against a rate, and the active ones are applied in shuffled order, so a sample is seen under a different corruption combination every epoch without storing anything offline. — For Mova (learn): Parser training could perturb sentences on the fly in label-preserving ways, for example swapping names, numbers or synonymous lemmas, with random on/off and ordering. This reduces overfitting to the exact surface forms in small treebanks.
  • I2. A generator (TTS) is trained specifically to widen speaker diversity, the dimension the real data lacks, rather than merely to add more examples of what already exists. — For Mova (back): The graph → text generator could produce paraphrases of known meaning graphs in varied styles to cover constructions missing from gold data. The generated pairs would then be fed to the learn stage as silver data.
  • I3. An explicit noise/NULL class lets the classifier say "nothing relevant here" instead of forcing a choice among the real labels. — For Mova (answer): Readers answering tests could be given an explicit abstain outcome when no frame in the text world supports any option. This outcome could also serve as a negative-control check during evaluation.
  • I4. A small systematic grid over width, depth and heads showed that depth and width pay off while extra heads are redundant. This is a cheap way to size a model under a parameter budget. — For Mova (tree): A similar compact ablation of gparse dimensions could find the smallest transformer parser that keeps LAS. The goal would be to avoid adding attention heads that bring no gain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.16621 · arXiv 2303.16621

@misc{salhab2023araspot,
  title = {{AraSpot: Arabic Spoken Command Spotting}},
  author = {Salhab, Mahmoud and Harmanani, Haidar},
  year = {2023},
  doi = {10.48550/arxiv.2303.16621},
  eprint = {2303.16621},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.16621}
}

References

43 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Alwajeeh, Mahmoud Al-Ayyoub, and Ismail Hmeidi. 2014. On authorship authentication of Arabic articles. In 2014 5th International Conference on Information and Communication Systems (ICICS). 1–6. https://doi.org/10.1109/ IACS.2014.6841973 ↗
  2. Sercan O. Arik, Markus Kliegl, Rewon Child, Joel Hestness, Andrew Gibiansky, Chris Fougner, Ryan Prenger, and Adam Coates. 2017. Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting. https://doi.org/10.48550/ARXIV.1703.05390 ↗
  3. Mostafa Awaid, Sahar Fawzy, and Ahmed Kandil. 2014. Audio Search Based on Keyword Spotting in Arabic Language. International Journal of Advanced Computer Science and Applications 5 (01 2014), 128. https://doi.org/10.14569/IJACSA. 2014.050219 ↗
  4. Lina Benamer and Osama Alkishriwo. 2020. Database for Arabic Speech Commands Recognition. find
  5. Naaima Boudad, Rdouan Faizi, Oulad haj thami Rachid, and Raddouane Chiheb. 2017. Sentiment analysis in Arabic: A review of the literature. Ain Shams Engineering Journal 9 (07 2017). https://doi.org/10.1016/j.asej.2017.04.007 ↗
  6. Guoguo Chen, Oguz Yilmaz, Jan Trmal, Daniel Povey, and Sanjeev Khudanpur. 2013. Using proxies for OOV keywords in the keyword search task. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. 416–421. https://doi.org/10.1109/ASRU.2013.6707766 ↗
  7. Alice Coucke, Mohammed Chlieh, Thibault Gisselbrecht, David Leroy, Mathieu Poumeyrol, and Thibaut Lavril. 2018. Efficient keyword spotting using dilated convolutions and gating. https://doi.org/10.48550/ARXIV.1811.07684 ↗
  8. Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. 2017. Freesound Datasets: A Platform for the Creation of Open Audio Datasets. find
  9. Abdulkader Ghandoura, Farouk Hjabo, and Oumayma Al Dakkak. 2021. Building and benchmarking an Arabic Speech Commands dataset for small-footprint keyword spotting. Engineering Applications of Artificial Intelligence 102 (2021), 104267. https://doi.org/10.1016/j.engappai.2021.104267 ↗
  10. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. https://doi.org/10.48550/ARXIV.2005.08100 ↗ our card
  11. Marco Jeub, Magnus Schafer, and Peter Vary. 2009. A binaural room impulse response database for the evaluation of dereverberation algorithms. In 2009 16th International Conference on Digital Signal Processing. 1–5. https://doi.org/10. 1109/ICDSP.2009.5201259 ↗
  12. Chanwoo Kim, Ananya Misra, Kean Chin, Thad Hughes, Arun Narayanan, Tara Sainath, and Michiel Bacchiani. 2017. Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home. 379–383. http://www.isca-speech.org/archive/Interspeech_2017/pdfs/1510.PDF ↗
  13. Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur. 2017. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5220–5224. https://doi.org/10.1109/ICASSP.2017.7953152 ↗
  14. Jason Li, Ravi Gadde, Boris Ginsburg, and Vitaly Lavrukhin. 2018. Training Neural Speech Recognition Systems with Synthetic Speech Augmentation. https://doi.org/10.48550/ARXIV.1811.00707 ↗
  15. James Lin, Kevin Kilgour, Dominik Roblek, and Matthew Sharifi. 2020. Training Keyword Spotters with Limited and Synthesized Speech Data. https://doi.org/10.48550/ARXIV.2002.01322 ↗
  16. Assaf Hurwitz Michaely, Xuedong Zhang, Gabor Simko, Carolina Parada, and Petar Aleksic. 2017. Keyword spotting for Google assistant using contextual speech recognition. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). 272–278. https://doi.org/10.1109/ASRU.2017.8268946 ↗
  17. David Miller, Michael Kleber, Chia-Lin Kao, O. Kimball, Thomas Colthurst, Stephen Lowe, Richard Schwartz, and Herbert Gish. 2007. Rapid and accurate spoken term detection. Proc. Interspeech, 314–317. https://doi.org/10.21437/ Interspeech.2007-174 ↗
  18. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Interspeech 2019. ISCA. https://doi.org/10.21437/interspeech.2019-2680 ↗ our card
  19. Jinhwan Park, Sichen Jin, Junmo Park, Sungsoo Kim, Dhairya Sandhyana, Changheon Lee, Myoungji Han, Jungin Lee, Seokyeong Jung, Changwoo Han, and Chanwoo Kim. 2023. Conformer-Based on-Device Streaming Speech Recognition with KD Compression and Two-Pass Architecture. In 2022 IEEE Spoken Language Technology Workshop (SLT). 92–99. https://doi.org/10.1109/SLT54892.2023.10023291 ↗
  20. Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2017. Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning. https://doi.org/10.48550/ ARXIV.1710.07654 ↗
  21. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2018. WaveGlow: A Flow-based Generative Network for Speech Synthesis. https://doi.org/10.48550/ARXIV.1811.00002 ↗
  22. Anirudh Raju, Sankaran Panchapagesan, Xing Liu, Arindam Mandal, and Nikko Strom. 2018. Data Augmentation for Robust Keyword Spotting under Playback Interference. arXiv:1808.00563 [cs.CL] ↗
  23. J.R. Rohlicek, W. Russell, S. Roukos, and H. Gish. 1989. Continuous hidden Markov modeling for speaker-independent word spotting. In International Conference on Acoustics, Speech, and Signal Processing,. 627–630 vol.1. https://doi.org/ 10.1109/ICASSP.1989.266505 ↗
  24. R.C. Rose and D.B. Paul. 1990. A hidden Markov model based keyword recognition system. In International Conference on Acoustics, Speech, and Signal Processing. 129–132 vol.1. https://doi.org/10.1109/ICASSP.1990.115555 ↗
  25. Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro Moreno, Yonghui Wu, and Zelin Wu. 2019. Speech Recognition with Augmented Synthesized Speech. https://doi.org/10.48550/ARXIV.1909.11699 ↗
  26. Deokjin Seo, Heung-Seon Oh, and Yuchul Jung. 2021. Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting. IEEE Access 9 (2021), 80682–80691. https://doi.org/10.1109/ACCESS.2021.3078715 ↗
  27. Changhao Shan, Junbo Zhang, Yujun Wang, and Lei Xie. 2018. Attention-based End-to-End Models for Small-Footprint Keyword Spotting. https://doi.org/10.48550/ARXIV.1803.10916 ↗
  28. Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. 2017. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. https://doi.org/10.48550/ARXIV.1712.05884 ↗
  29. Yihua Shi, Guanglin Ma, Jin Ren, Haigang Zhang, and Jinfeng Yang. 2022. An End-to-End Conformer-Based Speech Recognition Model for Mandarin Radiotelephony Communications in Civil Aviation. In Biometric Recognition, Weihong Deng, Jianjiang Feng, Di Huang, Meina Kan, Zhenan Sun, Fang Zheng, Wenfeng Wang, and Zhaofeng He (Eds.). Springer Nature Switzerland, Cham, 335–347. find
  30. Connor Shorten and Taghi Khoshgoftaar. 2019. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data 6 (07 2019). https://doi.org/10.1186/s40537-019-0197-0 ↗
  31. Jose M. R. Sotelo, Soroush Mehri, Kundan Kumar, João Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio. 2017. Char2Wav: End-to-End Speech Synthesis. In International Conference on Learning Representations. find
  32. Ming Sun, David Snyder, Yixin Gao, Varun Nagaraja, Mike Rodehorst, Sankaran Panchapagesan, Nikko Ström, Spyros Matsoukas, and Shiv Vitaladevuni. 2017. Compressed Time Delay Neural Network for Small-footprint Keyword Spotting. In Interspeech 2017. https://www.amazon.science/publications/compressed-time-delay-neural-network-for-small-footprint-keyword-spotting ↗
  33. Igor Szoke, Miroslav Skacel, Ladislav Mosner, Jakub Paliesek, and Jan Cernocky. 2019. Building and evaluation of a real room impulse response dataset. IEEE Journal of Selected Topics in Signal Processing 13, 4 (aug 2019), 863–876. https://doi.org/10.1109/jstsp.2019.2917582 ↗
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. https://doi.org/10.48550/ARXIV.1706.03762 ↗
  35. Oriol Vinyals and Steven Wegmann. 2014. Chasing the metric: Smoothing learning algorithms for keyword detection. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 3301–3305. https://doi.org/ 10.1109/ICASSP.2014.6854211 ↗
  36. Dong Wang, Shaohe Lv, Xiaodong Wang, and Xinye Lin. 2018. Gated Convolutional LSTM for Speech Commands Recognition. 669–681. https://doi.org/10.1007/978-3-319-93701-4_53 ↗
  37. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017. Tacotron: Towards End-to-End Speech Synthesis. https://doi.org/10.48550/ARXIV.1703.10135 ↗
  38. Pete Warden. 2018. Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. https://doi.org/10. 48550/ARXIV.1804.03209 ↗
  39. M. Weintraub. 1993. Keyword-spotting using SRI’s DECIPHER large-vocabulary speech-recognition system. In 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 2. 463–466 vol.2. https://doi.org/10.1109/ ICASSP.1993.319341 ↗
  40. Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. 2022. Image Data Augmentation for Deep Learning: A Survey. https://doi.org/10.48550/ARXIV.2204.08610 ↗
  41. Deokgyu Yun and Seung Ho Choi. 2022. Deep Learning-Based Estimation of Reverberant Environment for Audio Data Augmentation. Sensors 22, 2 (2022). https://doi.org/10.3390/s22020592 ↗
  42. Mengjun Zeng and Nanfeng Xiao. 2019. Effective Combination of DenseNet and BiLSTM for Keyword Spotting. IEEE Access 7 (2019), 10767–10775. https://doi.org/10.1109/ACCESS.2019.2891838 ↗
  43. Yang Zhang, Krishna C. Puvvada, Vitaly Lavrukhin, and Boris Ginsburg. 2023. Conformer-Based Target-Speaker Automatic Speech Recognition For Single-Channel Audio. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. https://doi.org/10.1109/icassp49357.2023.10095115 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salhab-2023-araspot-arabic-spoken-command-spotting/