Elamur Research
☰
home / papers / 2018 / Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces

Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1805.10190

Summary

The paper describes how Snips built a Spoken Language Understanding (SLU) system that runs fully offline on small IoT hardware. Because no audio or text from users leaves the device, the system is private by design. Mainstream ASR systems rely on very large neural networks and n-gram models, so they need the cloud. The authors instead designed compact models for both stages of the pipeline. The acoustic model is a hybrid NN/HMM of about 10MB, built on Kaldi. To make it robust without real user recordings, they simulate noisy and reverberant rooms. They also train the ASR language model and the NLU engine jointly, so both are consistent and specialized to each assistant's intents and slots. Since user data is never collected, training examples come from crowdsourcing combined with machine learning. The resulting engine runs in real time on a Raspberry Pi 3. On an in-house benchmark, its slot-filling accuracy is competitive with major cloud NLU services. This shows that private, embedded voice interfaces are a practical alternative to cloud assistants.

Key points

  • The reference hardware is a Raspberry Pi 3 (1.4 GHz CPU, 1GB RAM). Other supported targets include Linux boards (i.MX8M, DragonBoard 410c, Jetson TX2), Android 5+ and iOS 11+. Inference code was rewritten in Rust for speed and portability.
  • The acoustic model is a custom version of Kaldi's s5 recipe:
    • It uses 40 MFCC features with 25ms windows every 10ms.
    • A speaker-adaptive GMM-HMM bootstraps alignments for a time-delay neural network.
    • Training data for the models ranges from several hundred up to several thousand hours of speech.
  • Far-field and noisy conditions are simulated by re-recording training audio in thousands of virtual rooms, with random microphone and speaker placement.
  • The appendix NLU benchmark (June 2017) compares Snips with Luis, Wit, API.ai and Alexa on 7 intents (e.g., PlayMusic, GetWeather, BookRestaurant). It reports slot precision, recall and F1 with 70 and 2000 training utterances.
  • Adoption and openness: the NLU component is open source. Since mid-2017, over 13,000 developers have built more than 23,000 assistants. Full support covers English, French and German, with NLU-only support for Spanish and Korean.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

321 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 316 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1805.10190 · arXiv 1805.10190

@misc{coucke2018snips,
  title = {{Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces}},
  author = {Coucke, Alice and Saade, Alaa and Ball, Adrien and Bluche, Theodore and Caulier, Alexandre and Leroy, David and Doumouro, Clement and Gisselbrecht, Thibault and Caltagirone, Francesco and Lavril, Thibaut and Primet, Mael and Dureau, Joseph},
  year = {2018},
  doi = {10.48550/arxiv.1805.10190},
  eprint = {1805.10190},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.10190}
}

References

59 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Petar Aleksic, Mohammadreza Ghodsi, Assaf Michaely, Cyril Allauzen, Keith Hall, Brian Roark, David Rybach, and Pedro Moreno. Bringing contextual information to google speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  2. Cyril Allauzen, Michael Riley, and Johan Schalkwyk. A generalized composition algorithm for weighted finite-state transducers. In Tenth Annual Conference of the International Speech Communication Association, 2009. find
  3. Cyril Allauzen, Michael Riley, and Johan Schalkwyk. Filters for efficient composition of weighted finite-state transducers. In International Conference on Implementation and Application of Automata, pages 28–38. Springer, 2010. find
  4. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. Openfst: A general and efficient weighted finite-state transducer library. In International Conference on Implementation and Application of Automata, pages 11–23. Springer, 2007. find
  5. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4945–4949. IEEE, 2016. find
  6. Tobias Bocklet, Andreas Maier, Josef G Bauer, Felix Burkhardt, and Elmar Noth. Age and gender recognition for telephone applications based on gmm supervectors and support vector machines. In Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. IEEE International Conference on, pages 1605–1608… find
  7. Tom Bocklisch, Joey Faulker, Nick Pawlowski, and Alan Nichol. Rasa: Open source language understanding and dialogue management. arXiv preprint arXiv:1712.05181, 2017. — [arXiv:1712.05181](https://arxiv.org/abs/1712.05181) find
  8. Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, and Manfred Langen. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 174–185, 2017. find
  9. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479, 1992. find
  10. Ann Cavoukian. Privacy by design. Take the challenge. Information and privacy commissioner of Ontario, Canada, 2009. find
  11. Ciprian Chelba, Dan Bikel, Maria Shugrina, Patrick Nguyen, and Shankar Kumar. Large scale language modeling in automatic speech recognition. arXiv preprint arXiv:1210.8440, 2012. — [arXiv:1210.8440](https://arxiv.org/abs/1210.8440) find
  12. Ciprian Chelba, Johan Schalkwyk, Thorsten Brants, Vida Ha, Boulos Harb, Will Neveitt, Carolina Parada, and Peng Xu. Query language modeling for voice search. In Spoken Language Technology Workshop (SLT), 2010 IEEE, pages 127–132. IEEE, 2010. find
  13. Facebook. Language, engine, and tooling for expressing, testing, and evaluating composable language rules on input strings. GitHub repository, https://github.com/facebook/duckling, 2017. ↗
  14. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pages 6645–6649. IEEE, 2013. find
  15. Dilek Hakkani-Tür, Frédéric Béchet, Giuseppe Riccardi, and Gokhan Tur. Beyond asr 1-best: Using word confusion networks in spoken language understanding. Computer Speech & Language, 20(4):495–514, 2006. find
  16. Melinda Hamill, Vicky Young, Jennifer Boger, and Alex Mihailidis. Development of an automated speech recognition interface for personal emergency response systems. Journal of NeuroEngineering and Rehabilitation, 6(1):26, 2009. find
  17. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine… find
  18. Axel Horndasch, Caroline Kaufhold, and Elmar Nöth. How to add word classes to the kaldi speech recognition toolkit. In International Conference on Text, Speech, and Dialogue, pages 486–494. Springer, 2016. find
  19. Panagiotis G Ipeirotis, Foster Provost, Victor S Sheng, and Jing Wang. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28(2):402–441, 2014. find
  20. Hui Jiang. Confidence measures for speech recognition: A survey. Speech communication, 45(4):455–470, 2005. find
  21. Slava Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE transactions on acoustics, speech, and signal processing, 35(3):400– 401, 1987. find
  22. Chanwoo Kim, Ananya Misra, Kean Chin, Thad Hughes, Arun Narayanan, Tara Sainath, and Michiel Bacchiani. Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in google home. Proc. INTERSPEECH. ISCA, 2017. find
  23. Anjishnu Kumar, Arpit Gupta, Julian Chan, Sam Tucker, Bjorn Hoffmeister, and Markus Dreyer. Just ask: Building an architecture for extensible self-service spoken language understanding. arXiv preprint arXiv:1711.00549, 2017. — [arXiv:1711.00549](https://arxiv.org/abs/1711.00549) find
  24. John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001. find
  25. Marc Langheinrich. Privacy by design—principles of privacy-aware ubiquitous systems. In International conference on Ubiquitous Computing, pages 273–291. Springer, 2001. find
  26. Benjamin Lecouteux, Michel Vacher, and François Portet. Distant speech recognition in a smart home: Comparison of several multisource asrs in realistic conditions. In Interspeech 2011 Florence, pages 2273–2276, 2011. find
  27. Percy Liang. Semi-supervised learning for natural language. PhD thesis, Massachusetts Institute of Technology, 2005. find
  28. Lidia Mangu, Eric Brill, and Andreas Stolcke. Finding consensus in speech recognition: word error minimization and other applications of confusion networks. Computer Speech & Language, 14(4):373–400, 2000. find
  29. Nicholas D Matsakis and Felix S Klock II. The rust language. In ACM SIGAda Ada Letters, volume 34, pages 103–104. ACM, 2014. find
  30. Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, et al. Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):530–5… find
  31. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In Interspeech, pages 3771–3775, 2013. find
  32. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. find
  33. Abdel-rahman Mohamed, George E Dahl, and Geoffrey Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, 2012. find
  34. Mehryar Mohri, Fernando Pereira, and Michael Riley. Weighted finite-state transducers in speech recognition. Departmental Papers (CIS), page 11, 2001. find
  35. Mehryar Mohri, Fernando Pereira, and Michael Riley. Speech recognition with weighted finite-state transducers. In Springer Handbook of Speech Processing, pages 559–584. Springer, 2008. find
  36. Josef Robert Novak, Nobuaki Minematsu, and Keikichi Hirose. Phonetisaurus: Exploring grapheme-to-phoneme conversion with joint n-gram models in the wfst framework. Natural Language Engineering, 22(6):907–938, 2016. find
  37. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. Improved part-of-speech tagging for online conversational text with word clusters. Association for Computational Linguistics, 2013. find our card
  38. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE, 2015. find
  39. Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. A time delay neural network architecture for efficient modeling of long temporal contexts. In Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  40. Vijayaditya Peddinti, Yiming Wang, Daniel Povey, and Sanjeev Khudanpur. Low latency acoustic modeling using temporal convolution and lstms. IEEE Signal Processing Letters, 25(3):373–377, 2018. find
  41. Bastian Pfleging, Stefan Schneegass, and Albrecht Schmidt. Multimodal interaction in the car: combining speech and gestures on the steering wheel. In Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pages 155– 162. ACM, 2012. find
  42. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, number EPFL-CONF-192584. IEEE S… find
  43. Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. Purely sequence-trained neural networks for asr based on lattice-free mmi. In Interspeech, pages 2751–2755, 2016. find
  44. Rohit Prabhavalkar, Ouais Alsharif, Antoine Bruguier, and Lan McGraw. On the compression of recurrent neural networks with an application to lvcsr acoustic modeling for embedded speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 5970… find
  45. Christian Raymond and Giuseppe Riccardi. Generative and discriminative algorithms for spoken language understanding. In Eighth Annual Conference of the International Speech Communication Association, 2007. find
  46. Marta Sabou, Kalina Bontcheva, and Arno Scharl. Crowdsourcing research opportunities: lessons from natural language processing. In Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies, page 17. ACM, 2012. find
  47. Johan Schalkwyk, Doug Beeferman, Françoise Beaufays, Bill Byrne, Ciprian Chelba, Mike Cohen, Maryam Kamvar, and Brian Strope. “your word is my command”: Google search by voice: a case study. In Advances in speech recognition, pages 61–90. Springer, 2010. find
  48. Snips Team. Rustling, Rust implementation of Duckling. GitHub repository, https://github. com/snipsco/rustling, 2018. find
  49. Snips Team. Snips NLU rust, Snips NLU Rust implementation. GitHub repository, https: //github.com/snipsco/snips-nlu-rs, 2018. find
  50. Snips Team. Snips NLU, Snips Python library to extract meaning from text. GitHub repository, https://github.com/snipsco/snips-nlu, 2018. ↗
  51. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing, pages 254–263. Association for Computational Linguistics,… find
  52. Maksim Tkachenko and Andrey Simanovsky. Named entity recognition: Exploring features. In KONVENS, pages 118–127, 2012. find
  53. Gökhan Tür, Anoop Deoras, and Dilek Hakkani-Tür. Semantic parsing using word confusion networks with conditional random fields. In INTERSPEECH, pages 2579–2583, 2013. find
  54. Ye-Yi Wang and Alex Acero. Discriminative models for spoken language understanding. In Ninth International Conference on Spoken Language Processing, 2006. find
  55. Yiming Wang, Vijayaditya Peddinti, Hainan Xu, Xiaohui Zhang, Daniel Povey, and Sanjeev Khudanpur. Backstitch: Counteracting finite-sample bias via negative steps. NIPS (submitted), 2017. find
  56. Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256, 2016. — [arXiv:1610.05256](https://arxiv.org/abs/1610.05256) find
  57. Haihua Xu, Daniel Povey, Lidia Mangu, and Jie Zhu. Minimum bayes risk decoding and system combination based on a recursion for edit distance. Computer Speech & Language, 25(4):802–828, 2011. find
  58. Roman V Yampolskiy and Venu Govindaraju. Behavioural biometrics: a survey and classification. International Journal of Biometrics, 1(1):81–113, 2008. find
  59. Dong Yu, Jinyu Li, and Li Deng. Calibration of confidence measures in speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 19(8):2461–2473, 2011. find