Elamur Research⌕
☰
home / papers / 2024 / To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

Paper · 2024 · Abdul Waheed, Karima Kadaoui and Muhammad Abdul-Mageed

Summary

Multilingual speech recognisers such as Whisper and SeamlessM4T are large, costly to run, and mostly tested on Modern Standard Arabic. Their behaviour on dialects is therefore poorly understood. The authors address both problems. They distil Whisper-large-v2 into smaller Arabic-specific students, following the pseudo-labelling recipe of Gandhi et al. (2023). The teacher transcribes unlabelled audio, weak transcripts are discarded with a WER cutoff, and the student is trained on two signals: the surviving labels and the teacher's token distributions (KL term). For evaluation they assemble ten test sets, including several Common Voice versions and a new human-transcribed corpus of five under-represented dialects drawn from TV series. The best student is 25–50% smaller than its teacher. Its overall WER is 45.0%, better than both SeamlessM4T-large-v2 (47.0%) and the teacher (55.1%), and it is the strongest model on the new dialect data (56.9%). An error analysis shows that systems with language-model decoding hallucinate more. The work suggests that compact distilled models can be both cheaper and more robust to linguistic variation.

Key points

  • Training objective: a weighted sum of KL divergence (weight 0.8) and cross-entropy on filtered pseudo-labels (weight 1.0), with the weights taken from Gandhi et al. (2023).
  • Students: four variants, trained on 100K or 500K segments (roughly 100 or 500 hours). About a quarter of the segments are removed by a deliberately loose WER threshold of 80.
  • New dialect corpus: covers ALG, JOR, PAL, UAE and YEM, with 10,567 utterances, 121,293 words and about 13.3 hours. None of it was seen by any model during training.
  • Headline results: 45.0% overall WER for the best student, against 47.0% for SeamlessM4T-large-v2 and 55.1% for Whisper-large-v2. JOR and PAL were the easiest dialects across systems.
  • Compute cost: training used more than 3000 A100 GPU hours, estimated at over 500 kg of CO2.

Interesting ideas

  • I1. Truly unseen evaluation data matters. A distilled model should be judged on data guaranteed to be absent from the teacher's training set, otherwise robustness claims may only reflect memorisation.
  • I2. The filter can stay loose on purpose. A permissive quality threshold on teacher outputs imitates a setting with almost no gold labels, so the recipe stays honest for low-resource conditions.
  • I3. The student learns from two signals. It copies the teacher's final outputs and also matches its full token distributions, so it learns which alternatives the teacher found plausible.
  • I4. A smaller student can beat its teacher. A compact specialised model can outperform the larger generalist that taught it, so distillation can serve as a way to specialise rather than only to compress.
  • I5. Error types are worth more than one score. Sorting errors into types, such as hallucinated content from LM-style decoding, explains why a system fails better than a single WER does.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Keep a held-out set of texts that no LLM teacher tier or witness parser has seen, and treat it as the decisive test for any silver-trained component.
  • I2 (learn): When filtering silver trees by witness agreement, report results at both strict and loose agreement thresholds to show how much the gains depend on gold-like supervision.
  • I3 (tree): gparse could be distilled from a larger parser using both the arg-max heads/labels and the teacher's soft head and label distributions, giving a smaller, faster parser with less loss.
  • I4 (tree): Distil a general multilingual parser into an English-only, UD-dialect-specific student and compare LAS against the teacher, not just against speed.
  • I5 (answer): Tag reader failures by type, such as invented entities versus missed ones, so that fluent but ungrounded answers are caught and traced to the responsible stage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Abdul Waheed, Karima Kadaoui and Muhammad Abdul-Mageed. 2024. To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.680

@inproceedings{waheed2024distill,
  title = {{To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation}},
  author = {Abdul Waheed, Karima Kadaoui and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.680},
  url = {https://doi.org/10.18653/v1/2024.acl-long.680}
}

References

73 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of Arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  2. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye MARBERT: deep bidirectional transformers for arabic. CoRR, abs/2101.01785. Advances in neural information processing systems, 33:12449–12460. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020. Toward micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876, Online. Association for Computational Linguistics. find
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020b. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  5. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2021. Masc: Massive arabic speech corpus. find
  6. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279– 284. IEEE.
  7. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2019a. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. find
  8. Ahmed Ali, Shammur Chowdhury, Amir Hussein, and Yasser Hifny. 2021. Arabic code-switching speech recognition using monolingual data.
  9. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019b. The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. find
  10. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic mgb-3.
  11. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. ↗
  12. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021a. Xls-r: Self-supervised cross-lingual speech representation learning at scale.
  13. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021b. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296. ↗
  14. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020a. wav2vec 2.0: A framework for self-supervised learning of speech representations.
  15. Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. 2023. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187. ↗
  16. Kaushal Santosh Bhogale, Sai Sundaresan, Abhigyan Raman, Tahir Javed, Mitesh M Khapra, and Pratyush Kumar. 2023. Vistaar: Diverse benchmarks and training sets for indian language asr. arXiv preprint arXiv:2305.15386. ↗
  17. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  18. William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211. ↗
  19. Heng-Jui Chang, Shu-Wen Yang, and Hung-yi Lee. 2021. Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit BERT. CoRR, abs/2110.01900. ↗
  20. Vamsikrishna Chemudupati, Marzieh S. Tahaei, Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, Mehdi Rezagholizadeh, Boxing Chen, and Tiago H. Falk. 2023. On the transferability of whisper-based representations for "in-the-wild" cross-task downstream speech applications. ArXiv, abs/2305.14546. ↗
  21. Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. 2022. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915–3924. PMLR. ↗
  22. Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021. Towards one model to rule all: Multilingual strategy for dialectal code-switching arabic asr.
  23. Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady ElSahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Peng Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Shang-Wen Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, M.L. Ramadan, Abinesh Ramakrishnan, Anna Sun, Ke M. Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bo Yu, Pierre Yves Andrews, Can Balioglu, Marta Ruiz Costa-jussà, Onur Çelebi, Maha Elbayad, Cynthia Gao, Francisco Guzm’an, Justine T. Kao, Ann Lee, Alexandre Mourachko, Juan Miguel Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. Seam-lessm4t: Massively multilingual&multimodal machine translation. find
  24. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979. ↗
  25. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. Fleurs: Few-shot learning evaluation of universal representations of speech.
  26. Kunal Dhawan, Dima Rekesh, and Boris Ginsburg. 2023. Towards training bilingual and code-switched speech recognition models from monolingual data sources. arXiv preprint arXiv:2306.08753. ↗
  27. Nawar Halabi et al. 2016. Arabic speech corpus. Oxford Text Archive Core Collection. find
  28. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. ↗
  29. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. ↗
  30. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. A comprehensive evaluation of quantization strategies for large language models. arXiv preprint arXiv:2402.16775. ↗
  31. Sheikh Musa Kaleem, Tufail Rouf, Gousia Habib, Brejesh Lall, et al. 2024. A comprehensive review of knowledge distillation in computer vision. arXiv preprint arXiv:2404.00936. ↗
  32. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
  33. Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. 2020. Compress: Self-supervised learning by compressing representations. CoRR, abs/2010.14713. ↗
  34. Thomas Palmeira Ferraz, Marcely Zanon Boito, Caroline Brun, and Vassilina Nikoulina. 2024. Multilingual distilwhisper: Efficient distillation of multi-task speech models via language-specific experts. find
  35. Ajinkya Kulkarni, Atharva Kulkarni, Sara Abedal-monem Mohammad Shatnawi, and Hanan Aldarmaki. 2023. Clartts: An open-source classical arabic text-to-speech corpus. arXiv preprint arXiv:2303.00069. ↗
  36. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. ↗
  37. Solomon Kullback and R. A. Leibler. 1951. On information and sufficiency. Annals of Mathematical Statistics, 22:79–86. ↗
  38. Sanchit Gandhi. 2024. Speculative decoding for 2x faster whisper inference. Hugging Face Blog. find
  39. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700. ↗
  40. Sanchit Gandhi, Patrick von Platen, and Alexander M. Rush. 2023. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. ↗
  41. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. ↗
  42. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. find
  43. Nizar Y Habash. 2022. Introduction to Arabic natural language processing. Springer Nature.
  44. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. ↗
  45. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978. ↗
  46. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. 2017a. Data-free knowledge distillation for deep neural networks. ↗
  47. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. 2017b. Data-free knowledge distillation for deep neural networks. CoRR, abs/1710.07535. ↗
  48. Weijian Luo. 2023. A comprehensive survey on knowledge distillation of diffusion models. ArXiv, abs/2304.04262. ↗
  49. Vimal Manohar, Pegah Ghahremani, Daniel Povey, and Sanjeev Khudanpur. 2018. A teacher-student learning approach for unsupervised domain adaptation of sequence-trained asr models. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 250– 257. find
  50. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. Qasr: Qcri aljazeera speech resource – a large scale annotated arabic speech corpus. ↗
  51. Jinjie Ni, Yukun Ma, Wen Wang, Qian Chen, Dianwen Ng, Han Lei, Trung Hieu Nguyen, Chong Zhang, Bin Ma, and Erik Cambria. 2023. Adaptive knowledge distillation between text and speech pre-trained models. OpenNMT. Ctranslate2: Fast inference engine for transformer models. find
  52. Jing Pan, Tao Lei, Kwangyoun Kim, Kyu J. Han, and Shinji Watanabe. 2022. Sru++: Pioneering fast recurrence with attention for speech recognition. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7872–7876. ↗
  53. Juan M Perero-Codosero, Fernando M Espinoza-Cuadros, and Luis A Hernández-Gómez. 2022. A comparison of hybrid and end-to-end asr systems for the iberspeech-rtve 2020 speech-to-text transcription challenge. Applied Sciences, 12(2):903. ↗
  54. Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, and Shinji Watanabe. 2024. End-to-end speech recognition: A survey. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:325–351. ↗
  55. Riefkyanov Pratama and Agit Amrullah. 2024. Analysis of whisper automatic speech recognition performance on low resource language. Jurnal Pilar Nusa Mandiri, 20:1–8. ↗
  56. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023a. Scaling speech technology to 1,000+ languages. ↗
  57. Dima Rekesh, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Juang, Oleksii Hrinchuk, Ankur Kumar, and Boris Ginsburg. 2023. Fast conformer with linearly scalable attention for efficient speech recognition. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. find
  58. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  59. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  60. Hang Shao, Wei Wang, Bei Liu, Xun Gong, Haoyu Wang, and Yanmin Qian. 2023. Whisper-kdq: A lightweight whisper via guided knowledge distillation and quantization for efficient asr. find
  61. Suwon Shon, Ahmed M. Ali, Younes Samih, Hamdy Mubarak, and James R. Glass. 2020. Adi17: A fine-grained arabic dialect identification dataset. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8244–8248. SYSTRAN. faster-whisper: Faster whisper transcription with ctranslate2. find
  62. Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023a. N-shot benchmarking of whisper on diverse arabic speech recognition. arXiv preprint arXiv:2306.02902. ↗
  63. Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023b. N-shot benchmarking of whisper on diverse arabic speech recognition.
  64. Hawau Olamide Toyin, Amirbek Djanibekov, Ajinkya Kulkarni, and Hanan Aldarmaki. 2023. Artst: Arabic text and speech transformer. arXiv preprint arXiv:2310.16621. ↗
  65. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  66. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023b. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516. ↗
  67. Abdul Waheed, Bashar Talafha, Peter Sullivan, Abdel-Rahim Elmadany, and Muhammad Abdul-Mageed. 2023. VoxArabica: A robust dialect-aware Arabic speech recognition system. In Proceedings of ArabicNLP 2023, pages 441–449, Singapore (Hybrid). Association for Computational Linguistics.
  68. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗
  69. Aiden Williams, Andrea Demarco, and Claudia Borg. 2023. The Applicability of Wav2Vec2 and Whisper for Low-Resource Maltese ASR . In Proc. 2nd Annual Meeting of the ELRA/ISCA SIG on Under-resourced Languages (SIGUL 2023), pages 39–43. find
  70. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. ↗
  71. Xiaoyu Yang, Qiujia Li, Chao Zhang, and Philip C. Woodland. 2023. Knowledge distillation from multiple foundation models for end-to-end speech recognition. ↗
  72. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. 2023a. Google usm: Scaling automatic speech recognition beyond 100 languages. ↗
  73. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. 2023b. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/waheed-2024-distill-or-not-distill-robustness/