Elamur Research⌕
☰
home / papers / 2020 / Sequence-level self-learning with multiple hypotheses

Sequence-level self-learning with multiple hypotheses

Paper · 2020 · Kenichi Kumatani, Dimitrios Dimitriadis, Yashesh Gaur et al.

Summary

The paper addresses self-training for end-to-end speech recognition when untranscribed audio is the only adaptation material. In this setting there is no larger teacher model and no parallel clean/noisy data, so the seed model's own output must act as the label. Relying on the single top hypothesis spreads recognition errors into training. The authors instead adapt an attention-based encoder-decoder (AED) on the N-best hypotheses from beam search. Each hypothesis is weighted by a softmax over its confidence score, which approximates the sequence-level knowledge distillation loss. They optimise this objective in two ways: by treating the weighted hypotheses as multiple labels (MLL), or by giving each hypothesis its own branch in a multi-task (MTL) network. The approach is tested on accent adaptation from American to British English, and also in a federated learning scenario. On British speech, WER drops from 14.55% to 10.36% compared with a baseline trained only on US English. This matters for on-device and federated settings, where running heavy teacher decoding is not practical.

Key points

  • Self-learning is framed as sequence-level KD in which the teacher is the seed model itself. The 1-best approximation (in the style of Kim and Rush) is replaced by a weighted sum over N-best hypotheses, using softmax-normalised confidence scores with temperature T = 1.
  • There are two MTL architectures. The first shares the encoder, attention and decoder and separates only the output layer per hypothesis. The second shares only the encoder and attention and gives each hypothesis its own decoder plus output layer. New branches are initialised from the main-task weights.
  • Training loop:
    • beam-search decode to get the N-best list;
    • adapt with MLL or MTL;
    • then train on the 1-best alone;
    • repeat until character error rate on validation data stops improving.
    • Sub-task branches are dropped at decoding time.
  • The AED uses Chorowski-style attention with a ReLU scoring function and is trained with scheduled sampling.
  • Main result: British-English WER falls from 14.55% to 10.36% relative to the US-only baseline. The method is also studied under federated learning.

Interesting ideas

  • I1. Instead of trusting a model's single best output as a pseudo-label, spread the training signal over several ranked alternatives, weighted by a softmax of their scores; this softens the damage of hard-decision errors in any self-training loop.
  • I2. Add temporary auxiliary output heads, one per alternative hypothesis, on a shared trunk, and throw them away at inference; the shared layers are pushed toward a representation compatible with all plausible readings at no deployment cost.
  • I3. Alternate a soft multi-hypothesis adaptation phase with a hard 1-best refinement phase, stopping as soon as a validation metric no longer improves; this gives a simple, controllable schedule for noisy self-learning.
  • I4. The authors warn that self-learning only helps when the seed model is already reasonably accurate on the new data, otherwise errors compound; checking seed accuracy before adapting is a cheap safeguard.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver data could keep the k-best parses from witness parsers, weighted by score or agreement, rather than only the consensus tree.
  • I2 (tree): during self-training, gparse could carry extra arc/label heads fed by second- and third-best parses and ship only the main head.
  • I3 (learn): self-play and repair-rule loops could use the same two-phase cycle with early stopping on held-out gold plus negative controls.
  • I4 (learn): before self-adapting the tagger or parser to a new genre such as fables, first measure seed accuracy on a small gold sample and skip adaptation if it is too low.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

Cite

Cite as Kenichi Kumatani, Dimitrios Dimitriadis, Yashesh Gaur, Robert Gmyr, Sefik Emre Eskimez, Jinyu Li and Michael Zeng. 2020. Sequence-level self-learning with multiple hypotheses. arXiv:2112.05826 · Proc. Interspeech 2020, page 3775-3779.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2020-2020 · arXiv 2112.05826

@misc{kumatani2020sequence,
  title = {{Sequence-level self-learning with multiple hypotheses}},
  author = {Kenichi Kumatani, Dimitrios Dimitriadis, Yashesh Gaur, Robert Gmyr, Sefik Emre Eskimez, Jinyu Li and Michael Zeng},
  year = {2020},
  doi = {10.21437/interspeech.2020-2020},
  eprint = {2112.05826},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2020-2020}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. C. Woodland, D. Pye, and M. J. F. Gales, “Iterative unsupervised adaptation using maximum likelihood linear regression,” in The 4th International Conference on Spoken Language Processing, Philadelphia, PA, USA, October 3-6, 1996. ISCA, 1996. find
  2. G. Zavaliagkos and T. Colthurst, “Utilizing untranscribed training data to improve performance,” in DARPA Broadcast News Transcription and Understanding Workshop, Landsdowne, 1998, pp. 301–305. find
  3. L. Lamel, J. luc Gauvain, and G. Adda, “Lightly supervised and unsupervised acoustic model training,” Computer Speech and Language, vol. 16, pp. 115–129, 2002. ↗
  4. F. Wessel and H. Ney, “Unsupervised training of acoustic models for large vocabulary continuous speech recognition,” IEEE Transactions on Speech and Audio Processing, vol. 13, no. 1, pp. 23–31, 2005. ↗
  5. Y. Huang, Y. Wang, and Y. Gong, “Semi-supervised training in deep learning acoustic model,” in Proc. Interspeech, September 2016. find
  6. J. Li, R. Zhao, J.-T. Huang, and Y. Gong, “Learning small-size dnn with output-distribution-based criteria,” in Proc. Interspeech, September 2014. find
  7. J. H. M. Wong and M. J. F. Gales, “Sequence student-teacher training of deep neural networks,” in Proc. Interspeech, September 2016. find
  8. J. Li, M. L. Seltzer, X. Wang, R. Zhao, and Y. Gong, “Large-scale domain adaptation via teacher-student learning,” in Proc. Interspeech, 2017, pp. 2386–2390. find
  9. R. M. Mun’im, N. Inoue, and K. Shinoda, “Sequence-level knowledge distillation for model compression of attention-based sequence-to-sequence speech recognition,” CoRR, vol. abs/1811.04531, 2018. ↗
  10. S. H. K. Parthasarathi, N. Sivakrishnan, P. Ladkat, and N. Strom, “Realizing petabyte scale acoustic modeling,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 2, pp. 422–432, 2019. ↗
  11. L. Mosner, M. Wu, A. Raju, S. H. K. Parthasarathi, K. Kumatani, S. Sundaram, R. Maas, and B. Hoffmeister, “Improving noise robustness of automatic speech recognition via parallel data and teacher-student learning,” in Proc. ICASSP, 2019, pp. 6475–6479. find
  12. Z. Meng, J. Li, Y. Zhao, and Y. Gong, “Conditional teacher-student learning,” CoRR, vol. abs/1904.12399, 2019. ↗
  13. J. Huang, E. Marcheret, K. Visweswariah, V. Libal, and G. Potamianos, The IBM Rich Transcription 2007 Speech-to-Text Systems for Lecture Meetings. Berlin, Heidelberg: Springer, 2007, vol. 4625. find
  14. K. Kumatani, T. Arakawa, K. Yamamoto, J. W. McDonough, B. Raj, R. Singh, and I. Tashev, “Microphone array processing for distant speech recognition: Towards real-world deployment,” in Proc. APSIPA ASC, 2012. find
  15. V. Manohar, P. Ghahremani, D. Povey, and S. Khudanpur, “A teacher-student learning approach for unsupervised domain adaptation of sequence-trained ASR models,” in 2018 IEEE Spoken Language Technology Workshop, SLT 2018, Athens, Greece, December 18-21, 2018. IEEE, 2018, pp. 250–257. find
  16. J. Konecny, B. McMahan, and D. Ramage, “Federated Optimization: Distributed Optimization Beyond the Datacenter,” arXiv preprint arXiv:1511.03575v1, 2015. ↗
  17. B. Jayaraman, L. Wang, D. Evans, and Q. Gu, “Distributed learning without distress: Privacy-preserving empirical risk minimization,” in Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 6343–6354. find
  18. D. Dimitriadis, K. Kumatani, R. Gmyr, Y. Gaur, and S. E. Eskimez, “A federated approach in training acoustic models,” in Proc. Interspeech, 2020. find
  19. ——, “Federated transfer learning with dynamic gradient aggregation,” vol. ArXiv: 2008.02452, 2020. ↗
  20. A. Graves, “Sequence transduction with recurrent neural networks,” CoRR, vol. abs/1211.3711, 2012. ↗
  21. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates, Inc., 2015, pp. 577–585. ↗
  22. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in ICASSP, 2016, pp. 4945–4949. find
  23. W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell,” CoRR, vol. abs/1508.01211, 2015. ↗
  24. A. Graves, S. Fernández, and F. Gomez, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in In Proceedings of the International Conference on Machine Learning, ICML 2006, 2006, pp. 369– 376. find
  25. R. Prabhavalkar, K. Rao, T. N. Sainath, B. Li, L. Johnson, and N. Jaitly, “A comparison of sequence-to-sequence models for speech recognition,” in Proc. Interspeech, 2017, pp. 939–943. find
  26. E. Battenberg, J. Chen, R. Child, A. Coates, Y. Gaur, Y. Li, H. Liu, S. Satheesh, A. Sriram, and Z. Zhu, “Exploring neural transducers for end-to-end speech recognition,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2017, Okinawa, Japan, December 16-20, 2017. IEEE, 2017, pp. 206– 213. find
  27. C.-C. Chiu, W. Han, Y. Zhang, R. Pang, S. Kishchenko, P. Nguyen, A. Narayanan, H. Liao, S. Zhang, A. Kannan, R. Prabhavalkar, Z. Chen, T. N. Sainath, and Y. Wu, “A comparison of end-to-end models for long-form speech recognition,” 2019, pp. 889–896. find
  28. G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” ArXiv, vol. abs/1503.02531, 2015. ↗
  29. Y. Kim and A. M. Rush, “Sequence-level knowledge distillation,” CoRR, vol. abs/1606.07947, 2016. ↗
  30. Z. Meng, J. Li, Y. Gaur, and Y. Gong, “Domain adaptation via teacher-student learning for end-to-end speech recognition,” in ASRU. IEEE, December 2019. ↗
  31. R. Caruana, “Multitask learning,” Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, PA 15213, 9 1997. find
  32. S. Ruder, “An overview of multi-task learning in deep neural networks,” ArXiv, vol. abs/1706.05098, 2017. ↗
  33. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” CoRR, vol. abs/1506.03099, 2015. ↗
  34. Q. Xie, E. H. Hovy, M.-T. Luong, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” ArXiv, vol. abs/1911.04252, 2019. ↗
  35. “Sox - sound exchange, audio manipulation tool,” Available at http://sox.sourceforge.net/ (2020/04/28). ↗
  36. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in Proc. Interspeech, 2019, pp. 2613–2617.
  37. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” Proc. ICASSP, pp. 5206–5210, 2015. find
  38. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1715–1725.
  39. N. Rossenbach, A. Zeyer, R. Schlüter, and H. Ney, “Generating synthetic audio data for attention-based speech recognition systems,” in Proc. ICASSP, 2020, pp. 7069–7073. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumatani-2021-sequence-level-self-learning-multiple/