Elamur Research⌕
☰
home / papers / 2024 / Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

Summary

The paper tackles speech recognition for languages that are treated here as low-resource because labelled audio is scarce and spoken varieties differ widely across regions. The main example is Arabic, but Russian and Portuguese are also studied. The authors fine-tune the multilingual Wav2Vec2-xlsr-53 model on Mozilla Common Voice data that is enlarged with acoustically altered copies of the original clips. The alterations are pitch shifting, added Gaussian noise and band-stop filtering. They score transcripts with both WER and CER and compare against plain fine-tuned Wav2Vec2 and against Whisper figures taken from the Whisper paper. Training on the combined augmented data lowers error rates in all three languages. The average relative gain is about 34% in WER and about 53% in CER, and Arabic shows the largest drop in character errors. The authors also claim the system copes better with diacritics. The work matters because it shows that cheap signal-level perturbations can partly make up for missing hours of transcribed speech in dialect-heavy languages.

Key points

  • Fine-tuning data was limited: about 17 hours each for Arabic and Portuguese and about 30 hours for Russian, all from Mozilla Common Voice. The backbone was Wav2Vec2-xlsr-53.
  • There were three augmentations (pitch shifting, Gaussian noise, band-stop filter), and the best Arabic result used all three together.
  • WER fell from 46.5 to 27.6% in Arabic, from 54.6 to 35.8% in Russian, and from 43.3 to 31.8% in Portuguese. Arabic CER dropped from 37.3 to 19.0%.
  • Reported Whisper WER on Common Voice was 90.9% for Arabic, 40.6% for Russian and 35.2% for Portuguese. Whisper's CER was not available, so the CER comparison is only against the Wav2Vec2 baseline.
  • Average relative improvement was 33.9% in WER and 53.2% in CER. Proposed future work includes speed and background-noise augmentation, studying how much augmented data is needed, and recordings that mix several languages.

Interesting ideas

  • I1. Instead of collecting more labelled examples, the authors multiply the existing ones with label-preserving distortions that mimic real-world variation, which is a cheap fix for any data-starved model.
  • I2. Each distortion is tried on its own and then all are fused, so the gain from combining them can be seen separately from the gain of any single trick.
  • I3. The paper reports a coarse metric (WER) alongside a finer one (CER), because the finer one exposes near-misses that the coarse one counts as total failures.
  • I4. The motivation is a mismatch between clean training sources (audiobooks) and spontaneous, dialect-rich real input, so training is pushed toward the messier distribution.
  • I5. One baseline (Whisper) is quoted from its original paper rather than rerun under identical conditions, and only for WER, which weakens the head-to-head claim.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Text-level analogues could be used to harden the tagger and lemmatizer against messy input, such as case changes, dropped punctuation, typos and British/American spellings, while keeping the gold tags unchanged.
  • I2 (learn): Repair rules and silver-data filters could be evaluated the same way, one at a time and then jointly, to show whether improvements add up or overlap.
  • I3 (tree): Parser evaluation could report LAS together with UAS and per-relation label accuracy, so that an attachment that is right but mislabelled is told apart from a fully wrong one.
  • I4 (tree): When gparse is trained on converted treebanks, informal or web-genre sentences could be deliberately mixed in, so the parser does not overfit to edited news-style English.
  • I5 (learn): This is a reminder to always rerun baselines on the same split and metrics, and to add negative controls, before claiming that one Mova component beats another.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Speech & spoken language Low-resource languages Semitic languages

 Institutions

🎓Ariel UniversityAriel · Israel8 works · 519 citations🎓Holon Institute of TechnologyHolon · Israel2 works · 4 citations

Cite

Cite as Or Haim Anidjar, Revital Marbel and Roi Yozevitch. 2024. Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages. arXiv:2501.00425.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.00425 · arXiv 2501.00425

@misc{anidjar2024whisper,
  title = {{Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages}},
  author = {Or Haim Anidjar, Revital Marbel and Roi Yozevitch},
  year = {2024},
  doi = {10.48550/arxiv.2501.00425},
  eprint = {2501.00425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.00425}
}

References

46 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Engin Avci and Zuhtu Hakan Akpolat. Speech recognition using a wavelet packet adaptive network based fuzzy inference system. Expert Systems with Applications, 31(3):495–503, 2006. find
  2. Toktam Zoughi, Mohammad Mehdi Homayounpour, and Mahmood Deypir. Adaptive windows multiple deep residual networks for speech recognition. Expert Systems with Applications, 139:112840, 2020. find
  3. Jinyu Li et al. Recent advances in end-to-end automatic speech recognition. APSIPA Transactions on Signal and Information Processing, 11(1), 2022. find
  4. Charissa Ann Ronao and Sung-Bae Cho. Human activity recognition with smartphone sensors using deep learning neural networks. Expert systems with applications, 59:235–244, 2016. find
  5. Shannon McCrocklin, Claire Fettig, and Simon Markus. Salukispeech: Integrating a new asr tool into students’ english pronunciation practice. Pronunciation in Second Language Learning and Teaching Proceedings, 12(1), 2022. find
  6. Paula Bräuer and Athanasios Mazarakis. How to design audio-gamification for language learning with amazon alexa?—a long-term field experiment. International Journal of Human–Computer Interaction, pages 1–18, 2022. find
  7. Jung-Woo Ha, Kihyun Nam, Jingu Kang, Sang-Woo Lee, Sohee Yang, Hyunhoon Jung, Eunmi Kim, Hyeji Kim, Soojin Kim, Hyun Ah Kim, et al. Clovacall: Korean goal-oriented dialog speech corpus for automatic speech recognition of contact centers. arXiv preprint arXiv:2004.09367, 2020. ↗
  8. Swati Bandi and Anil Kothari. Artificial intelligence: An asset for the financial sector. Impact of Artificial Intelligence on Organizational Transformation, pages 259–287, 2022. find
  9. E Sezgin and S D’Arcy. Editorial: Voice technology and conversational agents in health care delivery. front. Public Health, 10:887492, 2022. find
  10. Rachel Dorn. Dialect-specific models for automatic speech recognition of african american vernacular english. In Proceedings of the Student Research Workshop Associated with RANLP 2019, pages 16–20, 2019. find
  11. Keisuke Kinoshita, Tsubasa Ochiai, Marc Delcroix, and Tomohiro Nakatani. Improving noise robust automatic speech recognition with single-channel time-domain enhancement network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7009–7013. find
  12. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449–12460, 2020a. our card
  13. Paul Deléglise, Yannick Esteve, Sylvain Meignier, and Teva Merlin. Improvements to the lium french asr system based on cmu sphinx: what helps to significantly reduce the word error rate? In Tenth Annual Conference of the International Speech Communication Association, 2009. find
  14. Mahault Garnerin, Solange Rossato, and Laurent Besacier. Investigating the impact of gender representation in asr training data: a case study on librispeech. In 3rd Workshop on Gender Bias in Natural Language Processing, pages 86–92. Association for Computational Linguistics, 2021. find
  15. John Huehnergard and Na‘ama Pat-El. Introduction to the semitic languages and their history. In The Semitic Languages, pages 1–21. Routledge, 2019. find
  16. Chatrine Qwaider, Stergios Chatzikyriakidis, and Simon Dobnik. Can modern standard arabic approaches be used for arabic dialects? sentiment analysis as a case study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 40–50, 2019. find
  17. Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages A P REPRINT Ayman Alhelbawy, Mark Lattimer, Udo Kruschwitz, Chris Fox, and Massimo Poesio. An nlp-powered human rights monitoring platform. Expert Systems with Applications, 153:113365, 2020. our card
  18. Abbas Khosravani, Philip N Garner, and Alexandros Lazaridis. Modeling dialectal variation for swiss german automatic speech recognition. In Interspeech, pages 2896–2900, 2021. find
  19. Hamzah A Alsayadi, Salah Al-Hagree, Fahd A Alqasemi, and Abdelaziz A Abdelhamid. Dialectal arabic speech recognition using cnn-lstm based on end-to-end deep learning. In 2022 2nd International Conference on Emerging Smart Technologies and Applications (eSmarTA), pages 1–8. find
  20. Santiago-Omar Caballero-Morales and Felipe Trujillo-Romero. Evolutionary approach for integration of multiple pronunciation patterns for enhancement of dysarthric speech recognition. Expert Systems with Applications, 41(3): 841–852, 2014. find
  21. Wenxin Hou, Yue Dong, Bairong Zhuang, Longfei Yang, Jiatong Shi, and Takahiro Shinozaki. Large-scale end-to-end multilingual speech recognition and language identification with multi-task learning. Babel, 37(4k):10k, 2020. find
  22. Tapesh Kumar, Mehul Mahrishi, and Gaurav Meena. A comprehensive review of recent automatic speech summarization and keyword identification techniques. Artificial Intelligence in Industrial Applications, pages 111–126, 2022. find
  23. Hong Luo, Xiao Xie, Penghua Li, and Feiyang Xin. Data augmentation based unsupervised pre-training for low-resource speech recognition. In 2024 36th Chinese Control and Decision Conference (CCDC), pages 5007–5012, 2024. doi:10.1109/CCDC62350.2024.10587399. ↗
  24. Anuroop Sriram, Michael Auli, and Alexei Baevski. Wav2vec-aug: Improved self-supervised training with limited data. arXiv preprint arXiv:2206.13654, 2022. ↗
  25. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013. our card
  26. Timothy J. O’Shea, Johnathan Corgan, and T. Charles Clancy. Unsupervised representation learning of structured radio communication signals. In 2016 First International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE), pages 1–5, 2016. doi:10.1109/SPLIM.2016.7528397. ↗
  27. Thanh Toan Nguyen, Minh Tam Pham, Thanh Tam Nguyen, Thanh Trung Huynh, Quoc Viet Hung Nguyen, Thanh Tho Quan, et al. Structural representation learning for network alignment with self-supervised anchor links. Expert Systems with Applications, 165:113857, 2021. find
  28. Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020. find
  29. Saeed Shurrab and Rehab Duwairi. Self-supervised learning methods and applications in medical imaging analysis: A survey. PeerJ Computer Science, 8:e1045, 2022. find
  30. Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1476–1485, 2019. find
  31. Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning, 108(8):1329–1351, 2019. find
  32. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12449– 12460. Curran Associates, Inc., 2020b. URL https://proceedings.neurips.cc/paper/2020/file/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf. ↗ our card
  33. Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. AI Open, 2022. find
  34. Munehiko Sasajima, Yoshinobu Kitamura, Mitsuru Ikeda, and Ruchiro Mizoguchi. A representation language for behavior and function: Fbrl. Expert systems with applications, 10(3-4):471–479, 1996. find
  35. Theo Deschamps-Berger, Lori Lamel, and Laurence Devillers. Investigating transformer encoders and fusion strategies for speech emotion recognition in emergency call center conversations. In Companion Publication of the 2022 International Conference on Multimodal Interaction, pages 144–153, 2022. find
  36. HAZ Shahgir, Khondker Salman Sayeed, and Tanjeem Azwad Zaman. Applying wav2vec2 for speech recognition on bengali common voices dataset. arXiv preprint arXiv:2209.06581, 2022. ↗
  37. Felipe Farias, Wilmer Lobato, William Cruz, and Marcellus Amadeus. Bilingual asr model with language identification for brazilian portuguese and south-american spanish. 2022. find
  38. Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages A P REPRINT Jiri Malek, Jakub Jansky, Zbynek Koldovsky, Tomas Kounovsky, Jaroslav Cmejla, and Jindrich Zdansky. Target speech extraction: Independent vector extraction guided by supervised speaker identification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2295–2309, 2022. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P McCrae. Dravidiancodemix: Sentiment analysis and offensive language identification dataset for dravidian languages in code-mixed text. Language Resources and Evaluation, pages 1–42, 2022. Shimaa Ahmed, Ilia Shumailov, Nicolas Papernot, and Kassem Fawaz. Towards more robust keyword spotting for voice assistants. In 31st USENIX Security Symposium (USENIX Security 22), 2022. Samuel Thomas, Kartik Audhkhasi, and Brian Kingsbury. Transliteration based data augmentation for training multilingual asr acoustic models in low resource settings. In INTERSPEECH, pages 4736–4740, 2020. Joel Shor, Dotan Emanuel, Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, et al. Personalizing asr for dysarthric and accented speech with limited data. arXiv preprint arXiv:1907.13511, 2019. S Shahnawazuddin, Nagaraj Adiga, Hemant Kumar Kathania, and B Tarun Sai. Creating speaker independent asr system through prosody modification based data augmentation. Pattern Recognition Letters, 131:213–218, 2020. Hamzah Alsayadi, Abdelaziz Abdelhamid, Islam Hegazy, and Zaki Taha. Data augmentation for arabic speech recognition based on end-to-end deep learning. International Journal of Intelligent Computing and Information Sciences, 21(2):50–64, 2021. Mohammed Bakheet. Improving speech recognition for arabic language using low amounts of labeled data, 2021. Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit asr pre-training? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6533–6537. ↗ our card
  39. IEEE, 2021. Or Haim Anidjar, Yannick Estève, Chen Hajaj, Amit Dvir, and Itshak Lapidot. Speech and multilingual natural language framework for speaker change detection and diarization. Expert Systems with Applications, 213:119238, 2023. Or Haim Anidjar, Chen Hajaj, Amit Dvir, and Issachar Gilad. A thousand words are worth more than one recording: Nlp based speaker change point detection. arXiv preprint arXiv:2006.01206, 2020. Zhao Meng, Lili Mou, and Zhi Jin. Hierarchical rnn with static sentence-level attention for text-based speaker change detection. In Conference on Information and Knowledge Management, pages 2203–2206, 2017. Marek Hrúz and Zbyněk Zajíc. Convolutional neural network for speaker change detection in telephone speaker diarization system. In ICASSP, pages 4945–4949, 2017. Qingjian Lin, Ruiqing Yin, Ming Li, Hervé Bredin, and Claude Barras. Lstm based similarity measurement with spectral clustering for speaker diarization. arXiv preprint arXiv:1907.10393, 2019. Stephen H Shum, Najim Dehak, Réda Dehak, and James R Glass. Unsupervised methods for speaker diarization: An integrated and iterative approach. IEEE Transactions on Audio, Speech, and Language Processing, 21(10): 2015–2028, 2013. Anna Silnova, Niko Brümmer, Johan Rohdin, Themos Stafylakis, and Lukáš Burget. Probabilistic embeddings for speaker diarization. In Odyssey, 2020. Veton Këpuska and Gamal Bohouta. Comparing speech recognition systems (microsoft api, google api and cmu sphinx). Int. J. Eng. Res. Appl, 7(03):20–24, 2017. Nenny Anggraini, Angga Kurniawan, Luh Kesuma Wardhani, and Nashrul Hakiem. Speech recognition application for the speech impaired using the android-based google cloud speech api. TELKOMNIKA (Telecommunication Computing Electronics and Control), 16(6):2733–2739, 2018. Willie Walker, Paul Lamere, Philip Kwok, Bhiksha Raj, Rita Singh, Evandro Gouvea, Peter Wolf, and Joe Woelfel. Sphinx-4: A flexible open source framework for speech recognition, 2004. Bo Li, Shuo-yiin Chang, Tara N Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, and Yonghui Wu. Towards fast and accurate streaming end-to-end asr. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6069–6073. ↗
  40. IEEE, 2020. Tomohiro Nakatani. Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In Proc. find
  41. Interspeech, 2019. Aurish Hammad Hafeez, Khawaja Mohiuddin, and Sohaib Ahmed. Speaker-dependent live quranic verses recitation recognition system using sphinx-4 framework. In 17th IEEE International Multi Topic Conference 2014, pages 333–337. find
  42. Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages A P REPRINT Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022. Zhaobin Kuang, Chidubem G Arachie, Bangyong Liang, Pradyumna Narayana, Giulia DeSalvo, Michael S Quinn, Bert Huang, Geoffrey Downs, and Yang Yang. Firebolt: Weak supervision under weaker assumptions. In International Conference on Artificial Intelligence and Statistics, pages 8214–8259. ↗ our card
  43. PMLR, 2022. Minh Tran and Mohammad Soleymani. Towards privacy-preserving speech representation for client-side data sharing. arXiv preprint arXiv:2203.14171, 2022. Lili Guo, Longbiao Wang, Jianwu Dang, Eng Siong Chng, and Seiichi Nakagawa. Learning affective representations based on magnitude and dynamic relative phase information for speech emotion recognition. Speech Communication, 136:118–127, 2022. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. ↗ our card
  44. PMLR, 2023. Sílvia Grasiella Moreira Almeida, Frederico Gadelha Guimarães, and Jaime Arturo Ramírez. Feature extraction in brazilian sign language recognition based on phonological structure and using rgb-d sensors. Expert Systems with Applications, 41(16):7259–7271, 2014. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. Kelly Berkson, Samson Lotven, Peng Hlei Thang, Thomas Thawngza, Zai Sung, James C Wamsley, Francis Tyers, Kenneth Van Bik, Sandra Kübler, Donald Williamson, et al. Building a common voice corpus for laiholh (hakha chin). In Proceedings of the Workshop on Computational Methods for Endangered Languages, volume 2, 2019. Kavita Chachadi and SR Nirmala. Voice-based gender recognition using neural network. In Information and Communication Technology for Competitive Strategies (ICTCS 2020), pages 741–749. ↗ our card
  45. Springer, 2022. Anton Ragni, Kate M Knill, Shakti P Rath, and Mark JF Gales. Data augmentation for low resource languages. In INTERSPEECH 2014: 15th Annual Conference of the International Speech Communication Association, pages 810–814. International Speech Communication Association (ISCA), 2014. Bao Thai, Robert Jimerson, Dominic Arcoraci, Emily Prud’hommeaux, and Raymond Ptucha. Synthetic data augmentation for improving low-resource asr. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW), pages 1–9. find
  46. IEEE, 2019. Odette Scharenborg, Francesco Ciannella, Shruti Palaskar, Alan Black, Florian Metze, Lucas Ondel, and Mark Hasegawa-Johnson. Building an asr system for a low-research language through the adaptation of a high-resource language asr system: preliminary results. In Proc. Internat. Conference on Natural Language, Signal and Speech Processing (ICNLSSP), pages 26–30, 2017. Arman Kheirati Roonizi and Christian Jutten. Band-stop smoothing filter design. IEEE Transactions on Signal Processing, 69:1797–1810, 2021. Majed El Helou and Sabine Süsstrunk. Blind universal bayesian image denoising with gaussian noise level learning. IEEE Transactions on Image Processing, 29:4885–4897, 2020. Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Velimirović. Spice: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1118–1128, 2020. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Yosuke Higuchi, Keita Karube, Tetsuji Ogawa, and Tetsunori Kobayashi. Hierarchical conditional end-to-end asr with ctc and multi-granular subword units. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7797–7801. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anidjar-2024-whisper-turns-stronger-augmenting-wav2vec/