Elamur Research⌕
☰
home / papers / 2021 / Analysis of Data Augmentation Methods for Low-Resource Maltese ASR

Analysis of Data Augmentation Methods for Low-Resource Maltese ASR

Summary

The paper asks how to build a usable speech recogniser for Maltese when only about 7 hours of transcribed speech exist. Maltese is a good stress test because even filtered web text for it is tiny. The authors start with a small labelled corpus and add several cheap resources:

  • a little unlabelled Maltese audio,
  • a Maltese text corpus of roughly 250m tokens,
  • a crude diphone/concatenative TTS system,
  • large speech collections in Arabic, Italian and English, which are historically linked to Maltese.

They first set out their own five-way taxonomy of augmentation methods, because the definitions used in earlier work overlap. They then test three families alone and in combination:

  • unsupervised training: a baseline Maltese model transcribes speech that has no labels;
  • multilingual training: an end-to-end system is pretrained on language mixtures and then fine-tuned;
  • synthesized speech: TTS output is used as extra training data.

SpecAugment is also used. Stacking the methods lowers WER by 15% absolute, with no language model involved. The result shows that modest compute and curated, related-language data can stand in for web-scale pretraining.

Key points

  • The starting point is about 7 hours of labelled Maltese speech, plus a small amount of untranscribed Maltese audio and a text corpus of about 250m tokens.
  • The OSCAR 2022 release has only about 118k Maltese tokens across 2200 documents, close to the size of Maltese Wikipedia (2214 documents).
  • The proposed taxonomy has five types: semi-supervised, data perturbation, synthesized data, unsupervised, and multilingual training.
  • The donor languages are Arabic (the Semitic stratum), Italian (the Romance superstratum) and English (the adstratum). Speech in these languages is decoded with the Maltese baseline model, or mixed into pretraining before fine-tuning.
  • Combining the techniques gives a 15% absolute WER gain, measured without a language model.

Interesting ideas

  • I1. The authors define "semi-supervised" as noisy labels from humans and "unsupervised" as labels produced by a machine, which removes the overlap in earlier definitions. — For Mova (learn): label every silver tree by where it came from (human-noisy, witness-parser agreement, LLM teacher) so training and error analysis can weight or filter each source separately.
  • I2. A weak model for the target language transcribes material from related languages, and the result becomes extra training data despite its errors. — For Mova (learn): run the current parser over neighbouring genres or dialects of English, keep only the agreed output as silver data, and then fine-tune on gold.
  • I3. Low-quality synthetic speech from a simple diphone TTS still helps, so augmentation data does not need to be high fidelity to be worth using. — For Mova (back): text generated from meaning graphs, even if stilted, could be fed back as extra parser training input, because its gold structure is known by construction.
  • I4. Gains are reported with no language model, so the effect of the augmentation is not mixed up with the decoder's priors. — For Mova (learn): measure each improvement with repair rules and downstream priors switched off, as a control that isolates which component actually helped.
  • I5. Donor languages are chosen by the historical layers of the target language rather than just by size. — For Mova (tokens): Romance-derived English vocabulary could draw on morphology resources from Latinate languages to improve lemma and derivation analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.07793 · arXiv 2111.07793

@misc{williams2021analysis,
  title = {{Analysis of Data Augmentation Methods for Low-Resource Maltese ASR}},
  author = {Williams;, Andrea DeMarco; Carlos Mena; Albert Gatt; Claudia Borg; Aiden and Plas, Lonneke van der},
  year = {2021},
  doi = {10.48550/arxiv.2111.07793},
  eprint = {2111.07793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.07793}
}

References

62 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460. find our card
  2. C. Wang, J. Pino, and J. Gu, “Improving cross-lingual transfer learning for end-to-end speech recognition with speech translation,” 2020. find
  3. V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert, “Massively multilingual ASR: 50 languages, 1 model, 1 billion parameters,” in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 4751–4755. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-2831 ↗
  4. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NIPS 2020, 2020. find our card
  5. S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” in Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 3465–3469. [Online]. Available: https://doi.org/10.21437/Interspeech.2019-1873 ↗
  6. J. Kreutzer, I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O. Suarez, I. Orife, K. Ogueji, A. N. Rubungo, T. Q. Nguyen, M. Müller, A. Müller, S. H. Muhammad, N. Muhammad, A. Mnyakeni, J. Mirzakhalov, T. Matangira, C. Leong, N. Lawson, S. Kudugunta, Y. Jernite, M. Jenny, O. Firat, B. F. P. Dossou, S. Dlamini, N. de Silva, S. Çabuk Ballı, S. Biderman, A. Battisti, A. Baruwa, A. Bapna, P. Baljekar, I. A. Azime, A. Awokoya, D. Ataman, O. Ahia, O. Ahia, S. Agrawal, and M. Adeyemi, “Quality at a glance: An audit of web-crawled multilingual datasets,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 50–72, 2022. [Online]. Available: https://aclanthology.org/2022.tacl-1.4 ↗ our card
  7. L. Besacier, E. Barnard, A. Karpov, and T. Schultz, “Automatic speech recognition for under-resourced languages: A survey,” Speech Communication, vol. 56, pp. 85–100, 2014. find
  8. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” in Proceedings of the fourth ACM Conference on Fairness, Accountability, and Transparency (FAccT’21). Online: Association for Computing Machinery, 2021. find
  9. J. Abadji, P. O. Suarez, L. Romary, and B. Sagot, “Towards a Cleaner Document-Oriented Multilingual Crawled Corpus,” arXiv 2201.06642, 2022, arXiv: 2201.06642. [Online]. Available: http://arxiv.org/abs/2201.06642 ↗ our card
  10. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings ofNAACL-HLT 2019,. Minneapolis, MN: Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: http://arxiv.org/abs/1810.04805 ↗
  11. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The State and Fate of Linguistic Diversity and Inclusion in the NLP World,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’. Online: Association for Computational Linguistics, 2020, pp. 6282–6293. find our card
  12. C. D. H. Mena, A. Gatt, A. DeMarco, C. Borg, L. van der Plas, A. Muscat, and I. Padovani, “Masri-headset: A maltese corpus for speech recognition,” in Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 6381–6388. find our card
  13. A. Gatt and S. Čéplö, “Digital corpora and other electronic resources for Maltese,” in Proceedings of the International Conference on Corpus Linguistics. Lancaster, UK: University of Lancaster, 2013. find
  14. J. J. Camilleri, “A Computational Grammar and Lexicon for Maltese,” Master’s thesis, Chalmers University of Technology, Gothenburg, Sweden, 2013. find
  15. M. Borg, K. Bugeja, C. Vella, G. Mangion, and C. Gafà, “Preparation of a free-running text corpus for Maltese concatenative speech synthesis,” Perspectives on Maltese Linguistics, Studia Typologica, vol. 14, pp. 297– 318, 2014, berlin:Akademie Verlag. find
  16. J. Brincat, Maltese and other languages: A linguistic history of Malta. Malta: Midsea Books, 2011. find
  17. C. Borg and A. Gatt, “Morphological analysis for the Maltese language: The challenges of a hybrid system.” in Proceedings of the 3rd Arabic Natural Language Processing Workshop (WANLP’17). Valencia, Spain: Association for Computational Linguistics, 2017. [Online]. Available: http://aclweb.org/anthology/W17-1304 ↗ our card
  18. P. Y. Simard, D. Steinkraus, J. C. Platt et al., “Best practices for convolutional neural networks applied to visual document analysis.” in Icdar, vol. 3, 2003. find
  19. J. M. Ramirez, A. Montalvo, and J. R. Calvo, “A survey of the effects of data augmentation for automatic speech recognition systems,” in Iberoamerican Congress on Pattern Recognition. Springer, 2019, pp. 669–678. find
  20. A. Ragni, K. Knill, S. P. Rath, and M. Gales, “Data augmentation for low resource languages,” in INTERSPEECH, 2014, pp. 810–814. find
  21. X. Cui, V. Goel, and B. Kingsbury, “Data augmentation for deep neural network acoustic modeling,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 9, pp. 1469–1477, 2015. find
  22. N. Jaitly and G. E. Hinton, “Vocal tract length perturbation (vtlp) improves speech recognition,” in Proc. ICML Workshop on Deep Learning for Audio, Speech and Language, vol. 117, 2013. find
  23. M. J. Gales, P. Woodland, H. Y. Chan, D. Mrva, R. Sinha, S. E. Tranter et al., “Progress in the cu-htk broadcast news transcription system,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 5, pp. 1513–1525, 2006. find
  24. L. Lamel, J.-L. Gauvain, and G. Adda, “Lightly supervised and unsupervised acoustic model training,” Computer Speech & Language, vol. 16, no. 1, pp. 115–129, 2002. find
  25. R. Jimerson, K. Simha, R. Ptucha, and E. Prud’hommeaux, “Improving asr output for endangered language documentation,” in The 6th Intl. Workshop on Spoken Language Technologies for Under-Resourced Languages, 2018. find
  26. B. Thai, R. Jimerson, D. Arcoraci, E. Prud’hommeaux, and R. Ptucha, “Synthetic data augmentation for improving low-resource asr,” in 2019 IEEE Western New York Image and Signal Processing Workshop (WNY-ISPW). IEEE, 2019, pp. 1–9. find
  27. B. Vachhani, C. Bhat, and S. K. Kopparapu, “Data augmentation using healthy speech for dysarthric speech recognition.” in Interspeech, 2018, pp. 471–475. find
  28. N. Kanda, R. Takeda, and Y. Obuchi, “Elastic spectral distortion for low resource speech recognition with deep neural networks,” in 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. IEEE, 2013, pp. 309–314. find
  29. J. Billa, “Isi asr system for the low resource speech recognition challenge for indian languages.” in INTERSPEECH, 2018, pp. 3207–3211. find
  30. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019. ↗ our card
  31. N. Rossenbach, A. Zeyer, R. Schlüter, and H. Ney, “Generating synthetic audio data for attention-based speech recognition systems,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7069–7073. find
  32. A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking: Speech chain by deep learning,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 301–308. find
  33. A. Rosenberg, Y. Zhang, B. Ramabhadran, Y. Jia, P. Moreno, Y. Wu, and Z. Wu, “Speech recognition with augmented synthesized speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 996–1002. find
  34. M. J. Gales, A. Ragni, H. AlDamarki, and C. Gautier, “Support vector machines for noise robust asr,” in 2009 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 2009, pp. 205–210. find
  35. J. Lööf, C. Gollan, and H. Ney, “Cross-language bootstrapping for unsupervised acoustic model training: Rapid development of a polish speech recognition system,” in Tenth Annual Conference of the International Speech Communication Association, 2009. find
  36. Y. Qian, K. Yu, and J. Liu, “Combination of data borrowing strategies for low-resource lvcsr,” in 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. IEEE, 2013, pp. 404–409. find
  37. G. Zavaliagkos and T. Colthurst, “Utilizing untranscribed training data to improve performance,” in DARPA Broadcast News Transcription and Understanding Workshop, Landsdowne, 1998. find
  38. H. Cucu, A. Buzo, and C. Burileanu, “Unsupervised acoustic model training using multiple seed asr systems,” in Spoken Language Technologies for Under-Resourced Languages, 2014. find
  39. T. Kemp and A. Waibel, “Unsupervised training of a speech recognizer: Recent experiments,” in Sixth European Conference on Speech Communication and Technology, 1999. find
  40. F. Wessel and H. Ney, “Unsupervised training of acoustic models for large vocabulary continuous speech recognition,” IEEE Transactions on Speech and Audio Processing, vol. 13, no. 1, pp. 23–31, 2004. find
  41. D. Imseng, P. Motlicek, H. Bourlard, and P. N. Garner, “Using out-of-language data to improve an under-resourced speech recognizer,” Speech communication, vol. 56, pp. 142–151, 2014. find
  42. E. Yılmaz, H. v. d. Heuvel, and D. A. van Leeuwen, “Acoustic and textual data augmentation for improved asr of code-switching speech,” arXiv preprint arXiv:1807.10945, 2018. ↗
  43. K. M. Knill, M. J. Gales, S. P. Rath, P. C. Woodland, C. Zhang, and S.-X. Zhang, “Investigation of multilingual deep neural networks for spoken term detection,” in 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. IEEE, 2013, pp. 138–143. find
  44. N. T. Vu and T. Schultz, “Multilingual multilayer perceptron for rapid language adaptation between and across language families.” in Interspeech, 2013, pp. 515–519. find
  45. F. Grézl, M. Karafiát, and K. Veselỳ, “Adaptation of multilingual stacked bottle-neck neural network structure for new language,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 7654–7658. find
  46. Z. Tüske, P. Golik, D. Nolden, R. Schlüter, and H. Ney, “Data augmentation, feature combination, and multilingual neural networks to improve asr and kws performance for low-resource languages,” in Fifteenth Annual Conference of the International Speech Communication Association, 2014. find
  47. Z. Tüske, J. Pinto, D. Willett, and R. Schlüter, “Investigation on cross-and multilingual mlp features under matched and mismatched acoustical conditions,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 7349–7353. find
  48. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210. find
  49. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 279–284. find our card
  50. N. Habash, A. Soudi, and T. Buckwalter, “On arabic transliteration,” in Arabic computational morphology. Springer, 2007, pp. 15–22. find
  51. J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, “Jasper: An end-to-end convolutional neural acoustic model,” arXiv preprint arXiv:1904.03288, 2019. ↗
  52. O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al., “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019. ↗
  53. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. ↗
  54. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376. find
  55. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in Proceedings of Interspeech 2019. Graz, Austria: ISCA, 2019, pp. 2613–2617. [Online]. Available: http://arxiv.org/abs/1904.08779 ↗ our card
  56. X. Huang, A. Acero, H.-W. Hon, and R. Reddy, Spoken language processing: A guide to theory, algorithm, and system development. Prentice hall PTR, 2001. find
  57. D. Jurafsky, C. Wooters, J. Segal, A. Stolcke, E. Fosler, G. Tajchaman, and N. Morgan, “Using a stochastic context-free grammar as a language model for speech recognition,” in 1995 International Conference on Acoustics, Speech, and Signal Processing, vol. 1. IEEE, 1995, pp. 189–192. find
  58. R. L. Weide, “The CMU pronouncing dictionary,” 1998. [Online]. Available: http://svn.code.sf.net/p/cmusphinx/code/trunk/cmudict/ ↗
  59. C. Mena, “The ciempiess proper-names pronouncing dictionary,” in Corpus presented at OpenCor 2019 Conference, Guanajuato City, Mexico.[Available online at https://opencor.gitlab.io/corpora-list/], 2019. ↗
  60. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: self-supervised cross-lingual speech representation learning at scale,” CoRR, vol. abs/2111.09296, 2021. [Online]. Available: https://arxiv.org/abs/2111.09296 ↗ our card
  61. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and Ł. Kaiser, “Attention Is All You Need,” in Proceedings of the 31st Conference on Neural Informaton Processing Systems (NIPS’17), Long Beach, CA, 2017. find
  62. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised Cross-lingual Representation Learning at Scale,” in Proceedings ofthe 58th Annual Meeting ofthe Association for Computational Linguistics (ACL’20), 2020, pp. 8440–8451. Andrea DeMarco read for his undergraduate degree in Information Technology (majoring in Computer Science and Artificial Intelligence) from the University of Malta, which he followed with an M.Sc. in Computer Science and Artificial Intelligence from the same institution. Following this, he moved to the UK where he completed a Ph.D. at the Audio, Speech and Language Processing Laboratory of University of East Anglia, one of the major centers find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/demarco-2021-analysis-data-augmentation-methods-low/