Elamur Research⌕
☰
home / papers / 2019 / A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding

A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding

Summary

Spoken language understanding (SLU) in task-oriented dialogue combines two linked subtasks: intent detection, which classifies an utterance, and slot filling, which labels its tokens. The slots tend to depend on the intent. Earlier joint models either shared parameters only or passed intent information through gate mechanisms. Gates are hard to interpret, and a single utterance-level intent can corrupt every slot decision if it is wrong. Qin et al. instead adapt Stack-Propagation, previously used for POS tagging and parsing. In their joint model, the predicted intent feeds directly into the slot-filling decoder, and the whole pipeline stays differentiable. A self-attentive encoder predicts an intent at every token, and the utterance intent is chosen by voting. If some tokens get the wrong intent, the correct ones still guide their own slots, which limits error propagation. On the SNIPS and ATIS benchmarks the model reached state-of-the-art results. Replacing the encoder with fine-tuned BERT raised scores further. The work shows that explicit, token-level intent signals give a more interpretable and more accurate way to couple the two SLU subtasks.

Key points

  • Without BERT, the model scores 94.2 slot F1, 98.0 intent accuracy and 86.9 overall accuracy on SNIPS. On ATIS it scores 95.9, 96.9 and 86.5 respectively.
  • With BERT base as the encoder, results rise to 97.0 / 99.0 / 92.9 on SNIPS and 96.1 / 97.5 / 88.6 on ATIS. This slightly exceeds the BERT SLU baseline (Chen et al., 2019) in overall accuracy on both datasets.
  • Separate BERT models for intent detection alone (using [CLS]) and for slot filling alone did worse than the joint BERT version. This supports modelling the two tasks together.
  • Token-level intent voting works like an ensemble: it reduces prediction variance and keeps richer intent features for the slot decoder.
  • The code is released at github.com/LeePleased/StackPropagation-SLU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1214

@inproceedings{qin2019stack,
  title = {{A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding}},
  author = {Qin, Libo and Che, Wanxiang and Li, Yangming and Wen, Haoyang and Liu, Ting},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1214},
  url = {https://doi.org/10.18653/v1/d19-1214}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909. — [arXiv:1902.10909](https://arxiv.org/abs/1902.10909) find
  2. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv… find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  4. Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling. In Proc. of ACL. find our card
  5. Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proc. of NAACL. find our card
  6. Patrick Haffner, Gokhan Tur, and Jerry H Wright. 2003. Optimizing svms for complex call classification. In In Proc. of ICASSP. find
  7. Dilek Hakkani-Tür, Gökhan Tür, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional rnn-lstm. In Interspeech. find
  8. Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990. find
  9. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  10. Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. 2016. Stochastic multiple choice learning for training diverse deep ensembles. In NIPS. find
  11. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. arXiv preprint arXiv:1609.01454. — [arXiv:1609.01454](https://arxiv.org/abs/1609.01454) find
  12. Christian Raymond and Giuseppe Riccardi. 2007. Generative and discriminative algorithms for spoken language understanding. In Eighth Annual Conference of the International Speech Communication Association. find
  13. Ruhi Sarikaya, Geoffrey E Hinton, and Bhuvana Ramabhadran. 2011. Deep belief nets for natural language call-routing. In ICASSP. find
  14. Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In Proc. of AAAI. find our card
  15. Gokhan Tur and Renato De Mori. 2011. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons. find
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  17. Yu Wang, Yilin Shen, and Hongxia Jin. 2018. A bi-model based rnn semantic frame parsing model for intent detection and slot filling. In Proc. of ACL. find
  18. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. find
  19. Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014. Spoken language understanding using long short-term memory neural networks. In SLT. find
  20. Qingyu Yin, Yu Zhang, Wei-Nan Zhang, Ting Liu, and William Yang Wang. 2018. Deep reinforcement learning for chinese zero pronoun resolution. In Proc. of ACL. find our card
  21. Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019. Joint slot filling and intent detection via capsule neural networks. In Proc. of ACL. find
  22. Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spoken language understanding. In Proc. of IJCAI. find
  23. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In Proc. of ACL. find
  24. Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proc. of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2019-stack-propagation-framework-token-level/