Elamur Research⌕
☰
home / papers / 2023 / Large-scale Language Model Rescoring on Long-form Data

Large-scale Language Model Rescoring on Long-form Data

Paper · 2023 · Tongzhou Chen, Cyril Allauzen, Yinghui Huang et al.

Summary

The paper asks whether very large language models help speech recognition of long recordings, using 30-minute YouTube videos as the test material. The first-pass recognizer is a conformer with HAT factorization. When a segment ends, its lattice is rescored by an external LM. The internal-LM estimate is subtracted and the external score is added, each with its own tuned weight. The external LMs compared are T5 and MT5 (XL, XXL), PaLM (8B–540B), a 70M conformer LM and a 4.5B MaxEnt LM. For code-switched Indian English, MT5 was further fine-tuned on WEBDOC, a Hindi/Indian-English web and Wikipedia corpus transliterated to Latin script. Two engineering choices matter a lot. First, a decoder with limited label context merges hypotheses, so the lattice becomes a real graph instead of a tree. Second, the previous segments' 1-best output is fed to the scorer as context. Against a MaxEnt first-pass baseline, WER falls by up to 8% relative and Salient Term Error Rate by up to 30% relative. Gains from LLMs and from a conventional neural LM add up. The authors present this as the first study to scale LLMs to long-form ASR.

Key points

  • Test locales: en-us and en-in (code-switched with Hindi), built from long-form YouTube audio. A 5% slice of the test utterances served as a dev set for tuning hyperparameters.
  • Combined score: S_HAT(Y|X) − µ·S_ILM(Y) + ν·S_ELM(Y), where µ and ν are tunable weights.
  • T5 and MT5 were pre-trained with span corruption for 1M steps, then fine-tuned for 100K steps on a prefix-LM objective. MT5 got 300K more LM steps on WEBDOC (170M sentences, 2.9B tokens). PaLM was used as-is.
  • The first-pass decoder conditions only on the two previous word-pieces. Beam hypotheses that share this context collapse into one lattice state.
  • Results: up to 8% relative WER reduction and up to 30% relative STER reduction versus a MaxEnt first-pass baseline. Combining the LLM and the conformer LM gives additive gains.

Interesting ideas

  • I1. When adding an external scorer, subtract the model's own internal prior so that the same preference is not counted twice.
  • I2. Bounding the history a decoder conditions on makes hypotheses with the same recent context merge, so a fixed beam yields a graph that holds many more alternatives than a trie.
  • I3. Score each segment with the already-decided best output of the previous segments as context, rather than scoring segments in isolation.
  • I4. Besides the overall error rate, track errors only on salient terms, since those carry the meaning users care about.
  • I5. A huge general model and a small in-domain model can give complementary, additive improvements when interpolated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When gparse n-best parses are rescored with an external scorer, discount the parser's built-in bias toward frequent attachments before mixing in the external score.
  • I2 (tokens): The tagger/morphology beam could merge states that share their last two tags, giving a compact lattice of readings that later stages can rescore.
  • I3 (world): When interpreting a new sentence, condition its ambiguity resolution on the committed world state from earlier sentences, keeping each decision traceable to that context.
  • I4 (learn): Alongside LAS, report accuracy on content-bearing predicates and entities, which reveals whether gains actually reach frames and answers.
  • I5 (learn): A broad LLM teacher tier and a small domain-trained witness parser could be combined as separate weighted votes when building silver data, rather than picking only one.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tongzhou Chen, Cyril Allauzen, Yinghui Huang, Daniel Park, David Rybach, W. Ronny Huang, Rodrigo Cabrera, Kartik Audhkhasi, Bhuvana Ramabhadran, Pedro J. Moreno and Michael Riley. 2023. Large-scale Language Model Rescoring on Long-form Data. arXiv:2306.08133 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.08133 · arXiv 2306.08133

@misc{chen2023large,
  title = {{Large-scale Language Model Rescoring on Long-form Data}},
  author = {Tongzhou Chen, Cyril Allauzen, Yinghui Huang, Daniel Park, David Rybach, W. Ronny Huang, Rodrigo Cabrera, Kartik Audhkhasi, Bhuvana Ramabhadran, Pedro J. Moreno and Michael Riley},
  year = {2023},
  doi = {10.48550/arxiv.2306.08133},
  eprint = {2306.08133},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.08133}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020. ↗
  2. A. Wang and K. Cho, “Bert has a mouth, and it must speak: Bert as a markov random field language model,” arXiv preprint arXiv:1902.04094, 2019. [20] C. Allauzen, E. Variani, M. Riley, D. Rybach, and H. Zhang, “A hybrid seq-2-seq ASR design for on-device and server applications,” in Interspeech 2021, 2021, pp. 4044–4048. [21] T. N. Sainath et al., “An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,” in Interspeech, 2021, pp. 1777–1781. [22] K. Hu et al., “Improving deliberation by text-only and semi-supervised training,” arXiv preprint arXiv:2206.14716, 2022. ↗
  3. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [23] L. Xue et al., “mT5: A massively multilingual pre-trained text-to-text transformer,” in 2021 NAACL: Human Language Technologies, Jun. 2021.
  4. T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [24] A. Gulati et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. ↗
  5. A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311, 2022. [25] H. Liao, E. McDermott, and A. Senior, “Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription,” in ASRU, 2013, pp. 368– 373. ↗
  6. X. Zheng, C. Zhang, and P. C. Woodland, “Adapting gpt, gpt-2 and bert language models for speech recognition,” in 2021 IEEE ASRU, 2021, pp. 162–168. find
  7. J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, “Masked language model scoring,” in 2020 ACL, Jul. 2020.
  8. E. Variani et al., “Neural oracle search on n-best hypotheses,” in ICASSP, 2020, pp. 7824–7828. find
  9. S.-H. Chiu and B. Chen, “Innovative bert-based reranking language models for speech recognition,” in SLT. IEEE, 2021, pp. 266–271. find
  10. T. Hori, Y. Kubo, and A. Nakamura, “Real-time one-pass decoding with recurrent neural network language model for speech recognition,” in ICASSP. IEEE, 2014, pp. 6364–6368. find
  11. J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” arXiv preprint arXiv:1612.02695, 2016. ↗
  12. T. Hori, J. Cho, and S. Watanabe, “End-to-end speech recognition with word-based rnn language models,” in SLT. IEEE, 2018, pp. 389–396. find
  13. C. Peyser et al., “Improving tail performance of a deliberation e2e asr model using a large text corpus,” arXiv preprint arXiv:2008.10491, 2020. ↗
  14. A. Sriram, H. Jun, S. Satheesh, and A. Coates, “Cold fusion: Training seq2seq models together with language models,” arXiv preprint arXiv:1708.06426, 2017. ↗
  15. C. Gulcehre et al., “On using monolingual corpora in neural machine translation,” arXiv preprint arXiv:1503.03535, 2015. ↗
  16. C. Shan et al., “Component fusion: Learning replaceable language model component for end-to-end speech recognition system,” in ICASSP. IEEE, 2019, pp. 5361–5635. find
  17. A. Kannan et al., “An analysis of incorporating an external language model into a sequence-to-sequence model,” in ICASSP, 2018, pp. 1–5828. find
  18. E. Variani, D. Rybach, C. Allauzen, and M. Riley, “Hybrid autoregressive transducer (hat),” in ICASSP, 2020, pp. 6139– 6143. find
  19. E. McDermott, H. Sak, and E. Variani, “A density ratio approach to language model fusion in end-to-end automatic speech recognition,” in ASRU, 2019, pp. 434–441. [26] A. Narayanan et al., “Recognizing long-form speech using streaming end-to-end models,” in ASRU, 2019, pp. 920–927. [27] J. Emond, B. Ramabhadran, B. Roark, P. Moreno, and M. Ma, “Transliteration based approaches to improve code-switched speech recognition performance,” in SLT. IEEE, 2018, pp. 448–455. [28] G. Wenzek et al., “Ccnet: Extracting high quality monolingual datasets from web crawl data,” arXiv preprint arXiv:1911.00359, 2019. [29] R. Botros et al., “Tied & reduced rnn-t decoder,” arXiv preprint arXiv:2109.07513, 2021. [30] F. Biadsy, K. Hall, P. Moreno, and B. Roark, “Backoff inspired features for maximum entropy language models,” 2014. [31] F. Biadsy, M. Ghodsi, and D. Caseiro, “Effectively building tera scale maxent language models incorporating non-linguistic signals,” 2017. [32] A. Tripathi, H. Lu, H. Sak, and H. Soltau, “Monotonic recurrent neural network transducer and decoding strategies,” in ASRU, 2019, pp. 944—-948. [33] R. Zazo, T. N. Sainath, G. Simko, and C. Parada, “Feature learning with raw-waveform cldnns for voice activity detection,” in Interspeech, 2016, pp. 3668–3672. [34] R. Prabhavalkar et al., “Less is more: Improved rnn-t decoding using limited label context and path merging,” in ICASSP, 2021, pp. 5659–5663. [35] J. Kaplan et al., “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [36] J. Hoffmann et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022. [37] V. Ravi et al., “Improving accuracy of rare words for rnn-transducer through unigram shallow fusion,” arXiv preprint arXiv:2012.00133, 2020. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-large-scale-language-model-rescoring/