Elamur Research⌕
☰
home / papers / 2020 / SLURP: A Spoken Language Understanding Resource Package

SLURP: A Spoken Language Understanding Resource Package

Summary

The paper addresses spoken language understanding (SLU), that is, extracting the meaning of a voice command directly from audio. End-to-end (E2E) systems lack large and diverse corpora in which speech recordings carry structured semantic annotation. The authors present SLURP, an English-language package. Its core is about 72k recordings of commands to a home assistant, annotated at three levels: scenario, action and entities. First, AMT workers composed the command texts from prompts that were deliberately designed not to suggest specific words. Then more than a hundred participants read these texts aloud in a room approximating household conditions, with recording done by a far-field microphone array and a headset. Besides the data, the package includes pipeline baseline systems (ASR + NLU) and a new metric, SLU-F1, for entity evaluation. It gives credit for partially correct entity values and yields an error distribution suitable for analysis. The modular pipelines proved to be a strong baseline. By contrast, well-known SOTA E2E models failed to learn on these data. Thus, SLURP poses a considerably harder and lexically richer task for the field than previous corpora.

Key points

  • Size: 72,277 audio files, of which 34,603 were recorded with a headset and 37,674 with a far-field microphone. This amounts to 177 speakers, approximately 58 hours of audio and 17,181 sentences covering 18 scenarios, 46 actions and 55 entity types. There is also a separate version, SLURP-synth: 69,253 synthesized recordings voiced by 34 Google TTS voices.
  • Comparison with Snips and FSC: there are about 6 times more sentences than in Snips, and about 2.5 times more audio than in FSC. There are 9 times more domains, and the corpus is on average about 10 times lexically richer. SLURP also outperforms both corpora on the LS2, CVS1 and MSTTR measures. It falls behind only in average sentence length.
  • Quality control: recordings that could not be force-aligned with their text in Multi-ASR were removed. For the rest, the Multi-ASR and Google-ASR transcripts were compared with the prompt texts. For approximately 60% of the data, at least one of these systems achieved WER=0. Adding recordings in which all entities were recognized correctly (EntityWER=0) raises the share to approximately 73%.
  • Baselines: combinations of two NLU and two ASR systems were compared; HerMiT/Multi-ASR performed best. For entities, top-down decoding works better: first the scenario and action, then the entities. SLU-F1 correlates weakly with standard F1 (ρ = 0.2), although it ranks the systems in the same order.
  • E2E models: SOTA E2E models, including the model of Lugosch et al. (2019b), did not converge after retraining on SLURP. The resource is published on GitHub: pswietojanski/slurp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

86 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.588 · arXiv 2011.13205

@inproceedings{bastianelli2020slurp,
  title = {{SLURP: A Spoken Language Understanding Resource Package}},
  author = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.588},
  eprint = {2011.13205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.588}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. find our card
  2. Peter Bell, Joachim Fainberg, Ondrej Klejch, Jinyu Li, Steve Renals, and Pawel Swietojanski. 2020. Adaptation algorithms for speech recognition: An overview. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… find
  4. Jean Carletta. 2007. Unleashing the killer corpus: experiences in creating the multi-everything ami meeting corpus. Language Resources and Evaluation, 41(2):181–190. find
  5. Christopher Cieri, David Miller, and Kevin Walker. 2004. The fisher corpus: a resource for the next generations of speech-to-text. In LREC, volume 4, pages 69–71. find our card
  6. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt-agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-desi… find our card
  7. Michael Covington, Congzhou He, Cati Brown-Johnson, Lorina Naci, and John Brown. 2006. How complex is that sentence? a proposed revision of the rosenberg and abbeduto d-level scale. find
  8. Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-smith, David Pallett, Er Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: the ATIS-3 corpus. In in Proc. ARPA Human Language Technology Workshop ’92, Plainsboro, NJ, pages 43–48. Morgan Kaufman… find
  9. T. Desot, F. Portet, and M. Vacher. 2019. Slu for voice command in smart home: Comparison of pipeline and end-to-end approaches. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 822–829. find
  10. Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5467–5471, Florence, Italy. Association for Computation… find
  11. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ICASSP’92, page 517–520, USA. IEEE Computer Society. find
  12. Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro J. Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. 2018. From audio to semantics: Approaches to end-to-end spoken language understanding. 2018 IEEE Spoken Language Technology Workshop (SLT), pages 720– 726… find
  13. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The atis spoken language systems pilot corpus. In Proceedings of the Workshop on Speech and Natural Language, HLT ’90, page 96–101, USA. Association for Computational Linguistics. find
  14. Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volu… find
  15. Wendell Johnson. 1944. Studies in Language Behavior: I. A Program of Research. National Foreign Language Center Technical Reports. American Psychological Association, Psychological Monographs: General and Applied. find
  16. Elizabeth Shriberg. 1997. labeling project coder”s Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. 2017. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Proce… find
  17. Batia Laufer. 1994. The lexical profile of second language writing: Does it change over time? RELC Journal, 25(2):21–33. find
  18. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In 10th International Workshop on Spoken Dialogue Systems Technology. find
  19. Xiaofei Lu. 2009. Automatic measurement of syntactic complexity in child language acquisition. find
  20. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of esl learners’ oral narratives. The Modern Language Journal, 96(2):190–208. find
  21. Loren Lugosch, Brett Meyer, Derek Nowrouzezahrai, and Mirco Ravanelli. 2019a. Using speech synthesis to train end-to-end spoken language understanding models. abs/1910.09463. — [arXiv:1910.09463](https://arxiv.org/abs/1910.09463) find
  22. Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019b. Speech model pre-training for end-to-end spoken language understanding. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 201… find
  23. Davide Marino and Thomas Hain. 2011. An analysis of automatic speech recognition with multiple microphones. In INTERSPEECH. find
  24. O. Miksik, I. Munasinghe, J. Asensio-Cubero, S. Reddy Bethi, S-T. Huang, S. Zylfo, X. Liu, T. Nica, A. Mitrocsak, S. Mezza, R. Beard, R. Shi, R. Ng, P. Mediano, Z. Fountas, S-H. Lee, J. Med-vesek, H. Zhuang, Y. Rogers, and P. Swietojanski. 2020. Building proactive voice assistants: When and how (not… — [arXiv:2005.01322](https://arxiv.org/abs/2005.01322) find
  25. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics. find our card
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE. find
  27. Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. A time delay neural network architecture for efficient modeling of long temporal contexts. In Sixteenth Annual Conference of the International Speech Communication Association. find
  28. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, CONF. IEEE Signal Process… find
  29. Natalia Tomashenko, Antoine Caubrière, and Yannick Estève. 2019. Investigating Adaptation and Transfer Learning for End-to-End Spoken Language Understanding from Speech. In Interspeech 2019, pages 824–828, Graz, Austria. ISCA. find
  30. Vanzo, Bastianelli, and Lemon. 2019. Hierarchical multi-task natural language understanding for cross-domain conversational AI: HERMIT NLU. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 254–263, Stockholm, Sweden. Association for Computational Linguistics. find
  31. K. Wolfe-Quintero, S. Inagaki, and H.Y. Kim. 1998. Second Language Development in Writing: Measures of Fluency, Accuracy, & Complexity. National Foreign Language Center Technical Reports. Second Language Teaching & Curriculum Center, University of Hawaii at Manoa. find
  32. Su Zhu, Zijian Zhao, Tiejun Zhao, Chengqing Zong, and Kai Yu. 2019. Catslu: The 1st chinese audio-textual spoken language understanding challenge. In 2019 International Conference on Multimodal Interaction, ICMI ’19, page 521–525, New York, NY, USA. Association for Computing Machinery. find
  33. Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. 2016. Purely sequence-trained neural networks for asr based on lattice-free mmi. find
  34. Abhinav Rastogi, Xiaoxue Zang, Srinivas Kumar Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schema-guided dialogue state tracking task at dstc8. In AAAI Dialog System Technology Challenges Workshop. find
  35. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019… find our card
  36. Dmitriy Serdyuk, Yongqiang Wang, Christian Fue-gen, Anuj Kumar, Baiyang Liu, and Yoshua Bengio. 2018. Towards end-to-end spoken language understanding. CoRR, abs/1802.08395. — [arXiv:1802.08395](https://arxiv.org/abs/1802.08395) find
  37. Pawel Swietojanski, Jinyu Li, and Steve Renals. 2016. Learning hidden unit contributions for unsupervised acoustic model adaptation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(8):1450–1463. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/bastianelli-2020-slurp-spoken-language-understanding-resource/