Elamur Research⌕
☰
home / papers / 2019 / Joint Slot Filling and Intent Detection via Capsule Neural Networks

Joint Slot Filling and Intent Detection via Capsule Neural Networks

Paper · 2019 · Chenwei Zhang, Yaliang Li, Nan Du et al.

Summary

Voice assistants and similar NLU systems must make two linked decisions. They tag each word with a slot label, such as marking "Sungmin" as an artist. They also assign an intent to the whole utterance, such as AddToPlaylist. Earlier systems either ran these steps as a pipeline or used joint sequence-labeling models that compress the utterance into one vector, which ignores the natural word → slot → intent hierarchy.

The authors propose CAPSULE-NLU, a capsule network with three layers: WordCaps, SlotCaps and IntentCaps. WordCaps produces context-aware word vectors with a BiLSTM. Instead of CRF tagging, words are assigned to slot capsules through dynamic routing-by-agreement. The resulting slot representations are then routed to intent capsules. A new re-routing step feeds the predicted intent back into the word-to-slot routing to refine slot labels.

On two real-world datasets, the model beat alternative architectures and existing commercial NLU services. The work shows that modeling the semantic hierarchy explicitly lets slot filling and intent detection reinforce each other.

Key points

  • Architecture: CAPSULE-NLU has three capsule types.
    • WordCaps is a BiLSTM with word embeddings trained from scratch; the authors note that ELMo or BERT could be plugged in.
    • SlotCaps holds one capsule per slot type.
    • IntentCaps holds one capsule per intent label.
  • Routing method: Slot filling is not done by sequence labeling. Each word's prediction vector is routed to slots using the dynamic routing-by-agreement algorithm of Sabour et al. (2017), with a squash nonlinearity and iterative agreement updates.
  • Re-routing: The inferred intent representation re-enters the WordCaps–SlotCaps routing. This helps recover slots that the first routing pass missed.
  • Interpretability: Visualizations of routing weights show how extra iterations help. For example, the model groups "operetta for the theatre organ" as a single object_name span, and these slots then push the utterance toward the correct SearchCreativeWork intent.
  • Evaluation: The model was compared with joint baselines (e.g., slot-gated and attention-based RNN models) and with commercial NLU services on two real-world datasets. The supplied excerpt omits the experimental section, so dataset names and exact scores are not included here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓University of Illinois ChicagoChicago · United States38 works · 19,844 citations

Also on the byline: Alibaba Group (United States).

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan and Philip Yu. 2019. Joint Slot Filling and Intent Detection via Capsule Neural Networks. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1519

@inproceedings{zhang2019joint,
  title = {{Joint Slot Filling and Intent Detection via Capsule Neural Networks}},
  author = {Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan and Philip Yu},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1519},
  url = {https://doi.org/10.18653/v1/p19-1519}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yun-Nung Chen, Dilek Hakkani-Tür, Gökhan Tür, Jianfeng Gao, and Li Deng. 2016. End-to-end memory networks with knowledge carryover for multi-turn spoken language understanding. In INTER-SPEECH, pages 3245–3249. find
  2. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 551–561. our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  4. Jingjing Gong, Xipeng Qiu, Shaojing Wang, and Xuanjing Huang. 2018. Information aggregation via dynamic routing for sequence encoding. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2742–2752. find
  5. Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… our card
  6. Dilek Hakkani-Tür, Gökhan Tür, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional rnn-lstm. In Interspeech, pages 715–719. find
  7. Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In International Conference on Artificial Neural Networks, pages 44–51. Springer. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  9. Jian Hu, Gang Wang, Fred Lochovsky, Jian-tao Sun, and Zheng Chen. 2009. Understanding user’s query intent with wikipedia. In WWW, pages 471–480. ACM. find
  10. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, ICML 2001, pages 282–289. find
  11. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. Interspeech, pages 685–689. find
  12. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. our card
  13. Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In NIPS, pages 3859–3869. find
  14. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2017. Disan: Directional self-attention network for rnn/cnn-free language understanding. arXiv preprint arXiv:1709.04696. find
  15. Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2017. Deep semantic role labeling with self-attention. arXiv preprint arXiv:1712.01586. our card
  16. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31. find
  17. Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010. What is left to be understood in atis? In Spoken Language Technology Workshop (SLT), 2010 IEEE, pages 19–24. IEEE. find
  18. Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3090–3099. our card
  19. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In ASRU, pages 78–83. IEEE. find
  20. Chenwei Zhang, Nan Du, Wei Fan, Yaliang Li, Chun-Ta Lu, and Philip S Yu. 2017. Bringing semantic structures to user intent detection in online medical queries. In IEEE Big Data, pages 1019–1026. find
  21. Chenwei Zhang, Wei Fan, Nan Du, and Philip S Yu. 2016. Mining user intentions from medical queries: A neural network based heterogeneous jointly modeling approach. In WWW, pages 1373–1384. find
  22. Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, and Zhou Zhao. 2018. Investigating capsule networks with dynamic routing for text classification. arXiv preprint arXiv:1804.00538. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-joint-slot-filling-intent-detection/