Elamur Research⌕
☰
home / papers / 2018 / Zero-shot User Intent Detection via Capsule Neural Networks

Zero-shot User Intent Detection via Capsule Neural Networks

Paper · 2018 · Congying Xia, Chenwei Zhang, Xiaohui Yan et al.

Summary

Voice assistants keep gaining new skills, so intent classifiers often meet user intents with no annotated utterances yet. The authors study this zero-shot intent detection setting, where the emerging intents do not overlap with the training intents. They adapt capsule networks to text in a model called IntentCapsNet. Its SemanticCaps layer, a bidirectional LSTM with multi-head self-attention, extracts separate semantic features of an utterance, such as an action or an entity name. Its DetectionCaps layer merges these features through dynamic routing-by-agreement into one activation vector per intent. The zero-shot extension, IntentCapsNet-ZSL, builds Zero-shot DetectionCaps for unseen intents from three sources: the trained SemanticCaps, the routing learned on known intents, and similarities between intent label embeddings. It needs no label ontologies or hand-defined attributes. Experiments on SNIPS-NLU and CVA show that the models handle diversely phrased known intents better and can recognize new intents without labeled examples. Attention analyses also make the model's behaviour easier to interpret. The work matters because it lets dialog systems grow cheaply, without new annotation or full retraining.

Key points

  • The model has three capsule types: SemanticCaps (feature extraction), DetectionCaps (aggregation for known intents) and Zero-shot DetectionCaps (emerging intents). Word inputs are skip-gram embeddings.
  • At inference, vote vectors computed for existing intents are transferred to emerging ones. The transfer is weighted by label-embedding similarity, for example from PlayMusic to AddToPlaylist.
  • The model is evaluated on two real-world datasets, SNIPS-NLU and CVA. The CVA analysis covers 20 emerging intents.
  • On CVA, 13 of the 20 emerging intents have a high-variance similarity distribution (var(q_l) > 0.005), and all of these reach accuracy ≥ 0.83. A low variance does not always mean poor results.
  • Attention heads carry over across intents. The head that tracks the "play" action in PlayMusic also highlights rating actions in the unseen RateABook intent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States38 works · 19,844 citations🎓Jilin UniversityChangchun · China8 works · 5,204 citations

Also on the byline: Huawei Technologies (United States).

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang and Philip Yu. 2018. Zero-shot User Intent Detection via Capsule Neural Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1348

@inproceedings{xia2018zero,
  title = {{Zero-shot User Intent Detection via Capsule Neural Networks}},
  author = {Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang and Philip Yu},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1348},
  url = {https://doi.org/10.18653/v1/d18-1348}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. 2016. Synthesized classifiers for zero-shot learning. In CVPR, pages 5327–5336. find
  2. Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2016a. Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models. In ICASSP, pages 6045–6049. find
  3. Yun-Nung Chen, Dilek Hakkani-Tür, Gökhan Tür, Jianfeng Gao, and Li Deng. 2016b. End-to-end memory networks with knowledge carryover for multi-turn spoken language understanding. In IN-TERSPEECH, pages 3245–3249. find
  4. Emmanuel Ferreira, Bassam Jabaian, and Fabrice Lefevre. 2015a. Online adaptative zero-shot learning spoken language understanding using word-embedding. In ICASSP, pages 5321–5325. find
  5. Emmanuel Ferreira, Bassam Jabaian, and Fabrice Lefèvre. 2015b. Zero-shot semantic parser for spoken language understanding. In INTERSPEECH, pages 1403–1407. find
  6. Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. 2013. Devise: A deep visual-semantic embedding model. In NIPS, pages 2121–2129. find
  7. Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In ICANN, pages 44–51. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. representations in vector space. arXiv:1301.3781.
  9. Matthew B Hoy. 2018. Alexa, siri, cortana, and more: An introduction to voice assistants. Medical reference services quarterly, 37(1):81–88. find
  10. Jian Hu, Gang Wang, Fred Lochovsky, Jian-tao Sun, and Zheng Chen. 2009. Understanding user’s query intent with wikipedia. In WWW, pages 471–480. find
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. arXiv preprint Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In NIPS, pages 3859–3869.
  12. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  14. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In NIPS, pages 935– 943. find
  15. Anjishnu Kumar, Pavankumar Reddy Muddireddy, Markus Dreyer, and Björn Hoffmeister. 2017. Zero-shot learning across heterogeneous overlapping domains. In INTERSPEECH, volume 2017, pages 2914–2918. find
  16. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP, pages 1422– 1432.
  17. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pages 282–289. find
  18. Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2014. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453–465. find
  19. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR.
  20. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In INTERSPEECH, pages 685–689. find
  21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000–6010.
  22. IBM Watson Assistant. 2017. Defining intents. In https://console.bluemix.net/docs/ services/conversation/intents. html\#defining-intents. ↗
  23. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In ASRU, pages 78–83. find
  24. Majid Yazdani and James Henderson. 2015. A model of zero-shot learning of spoken language understanding. In EMNLP, pages 244–249. find
  25. Chenwei Zhang, Wei Fan, Nan Du, and Philip S Yu. 2016. Mining user intentions from medical queries: A neural network based heterogeneous jointly modeling approach. In WWW, pages 1373–1384. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2018-zero-shot-user-intent-detection/