Elamur Research⌕
☰
home / papers / 2024 / Towards a World-English Language Model for On-Device Virtual Assistants

Towards a World-English Language Model for On-Device Virtual Assistants

Paper · 2024 · Rricha Jalota, Lyan Verwimp, Markus Nussbaum-Thom et al.

Summary

Voice assistants usually ship a separate neural language model for each language, region and sometimes device. That multiplies training, testing and maintenance work. The authors replace the separate on-device English LMs for en_US, en_GB and en_IN with a single "World English" NNLM. The models are FOFE-based feedforward networks used in the first pass of a hybrid ASR pipeline. Two existing production designs, Mixture FOFE and Application-Dependent (AD) FOFE, are first trained on pooled multi-dialect data as baselines. The authors then add dialect-specific bottleneck adapters and test where to place them, three ways to train them, and a dual-adapter variant. Small adapters turn out to model dialect differences better than giving each dialect its own full sub-network. Building on this, they propose AD+CAA+DA, which combines:

  • shared layers,
  • per-application sub-networks,
  • a common application adapter,
  • dual (shared plus per-dialect) adapters.

It slightly lowers WER compared with the single-dialect models, especially on rare entities, and stays within on-device latency and memory budgets. One model can therefore stand in for three.

Key points

  • Training data: single-dialect models use 12B words with a 100k vocabulary. The multi-dialect model uses 36B words, sampled equally per dialect, with a 150k vocabulary that covers over 75% of each single-dialect vocabulary.
  • Three adapter training regimes are compared:
    • RI-A: the adapter starts near identity on a frozen pretrained model.
    • PT-A: the adapter is trained jointly during pretraining.
    • FT-A: the pretrained adapter is then fine-tuned on one dialect.
  • Evaluation covers three measures:
    • WER on three test sets: Assistant, STT and a TTS-synthesized Tail Entities set.
    • Model size in parameters.
    • On-device ASR processing latency, reported as the average and the P95.
  • Compared with the single-dialect baselines, AD+CAA+DA improves WER by about 1.63% relative on head-heavy sets and about 3.72% on tail entities.
  • Against the adapter-enhanced Mixture model, it shows a 27% average advantage on STT and runs 40% faster on the slowest 5% of Assistant queries.

Interesting ideas

  • I1. Specialize with lightweight adapters on a shared trunk rather than duplicating whole sub-networks per variety; the paper finds this both cheaper and more accurate.
  • I2. Dual adapters pair a common adapter, which learns what all varieties share, with a variety-specific one that captures differences, keeping the two kinds of knowledge separated.
  • I3. When the variety is known in advance from metadata, only the relevant sub-network is activated at inference, which saves time and memory without any guessing.
  • I4. Head-heavy test sets are complemented by a synthetic tail-entities test set, so gains on rare items are not hidden by the frequent ones.
  • I5. The joint vocabulary size is chosen by a coverage target against each per-variety vocabulary, trading coverage against training cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could share one encoder across UD treebanks and English varieties, with small per-treebank adapters that absorb annotation-dialect quirks instead of training separate parsers.
  • I2 (tree): a shared adapter plus per-genre adapters (fables, school texts, news) would make it inspectable which parsing decisions stem from genre-specific behavior.
  • I3 (tokens): the en crate could switch tagger and lemma rules by declared spelling variety (US/GB/IN) instead of mixing them, keeping each choice traceable.
  • I4 (learn): add a deliberately rare-vocabulary or rare-construction evaluation slice alongside the regular ones to catch regressions that averages mask.
  • I5 (tokens): pick lexicon and morphology coverage thresholds per source corpus explicitly, rather than taking a global top-N list.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AppleCupertino · United States20 works · 2,146 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Rricha Jalota, Lyan Verwimp, Markus Nussbaum-Thom, Amr Mousa, Arturo Argueta and Youssef Oualil. 2024. Towards a World-English Language Model for On-Device Virtual Assistants. arXiv:2403.18783.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/icassp48485.2024.10448018 · arXiv 2403.18783

@misc{jalota2024world,
  title = {{Towards a World-English Language Model for On-Device Virtual Assistants}},
  author = {Rricha Jalota, Lyan Verwimp, Markus Nussbaum-Thom, Amr Mousa, Arturo Argueta and Youssef Oualil},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10448018},
  eprint = {2403.18783},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icassp48485.2024.10448018}
}

References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Markus Nussbaum-Thom, Lyan Verwimp, and Youssef Oualil, “Application-agnostic language modeling for on-device ASR,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), July 2023, pp. 268–275. ↗
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, Eds., 2020. ↗
  3. Hemant Yadav and Sunayana Sitaram, “A survey of multilingual models for automatic speech recognition,” arXiv preprint arXiv:2202.12576, 2022. ↗
  4. Bo Li, Tara N. Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yanghui Wu, and Kanishka Rao, “Multi-dialect speech recognition with a single sequence-to-sequence model,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 4749–4753. find
  5. Vikas Joshi, Amit Das, Eric Sun, Rupesh R Mehta, Jinyu Li, and Yifan Gong, “Multiple Softmax Architecture for Streaming Multilingual End-to-End ASR Systems,” in Proceedings Interspeech, 2021, pp. 1767–1771. find
  6. Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert, “Massively multilingual asr: 50 languages, 1 model, 1 billion parameters,” arXiv preprint arXiv:2007.03001, 2020. ↗
  7. Genta Indra Winata, Guangsen Wang, Caiming Xiong, and Steven C. H. Hoi, “Adapt-and-adjust: Overcoming the long-tail problem of multilingual speech recognition,” in Interspeech, 2020. find
  8. Anjuli Kannan, Arindrima Datta, Tara N Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee, “Large-scale multilingual speech recognition with a streaming end-to-end model,” in Proceedings Interspeech, 2019. find
  9. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi, “Learning multiple visual domains with residual adapters,” Advances in neural information processing systems, vol. 30, 2017. ↗
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, “Parameter-efficient transfer learning for NLP,” in International Conference on Machine Learning (ICML), 2019, pp. 2790– 2799. ↗
  11. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder, “Mad-x: An adapter-based framework for multi-task cross-lingual transfer,” arXiv preprint arXiv:2005.00052, 2020. ↗
  12. Alex Graves, “Sequence transduction with recurrent neural networks,” in International Conference of Machine Learning (ICML) Workshop on Representation Learning, 2012. ↗
  13. Shiliang Zhang, Hui Jiang, Mingbin Xu, Junfeng Hou, and Li-Rong Dai, “The fixed-size ordinally-forgetting encoding method for neural network language models,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2015, pp. 495–500. find
  14. Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra, “A Primer on Pretrained Multilingual Language Models,” arXiv preprint arXiv:2107.00676, 2021. ↗
  15. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych, “AdapterFusion: Non-destructive task composition for transfer learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, Apr. 2021, pp. 487–503.
  16. Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe, “Lifting the curse of multilinguality by pre-training modular transformers,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, 2022, pp. 3479–3495. ↗
  17. Zhen Huang, Tim Ng, Leo Liu, Henry Mason, Xiaodan Zhuang, and Daben Liu, “SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6854–6858. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jalota-2024-towards-world-english-language-model/