Elamur Research⌕
☰
home / papers / 2020 / Efficient Dialogue State Tracking by Selectively Overwriting Memory

Efficient Dialogue State Tracking by Selectively Overwriting Memory

Summary

Dialogue state tracking (DST) records what a user wants during a task-oriented conversation as a set of slot-value pairs. Open vocabulary-based trackers remove the need for a predefined ontology, but most of them rebuild the full state at every turn, which wastes computation. The authors, from Clova AI at NAVER, treat the state as a fixed-size memory that only needs partial rewriting. In SOM-DST, a BERT-based encoder reads the previous and current turn utterances along with the previous state. It then assigns each slot one of four operations: CARRYOVER, DELETE, DONTCARE or UPDATE. A decoder generates values autoregressively, but only for slots marked UPDATE. Splitting the task this way gives each component a narrower job and makes the decoder's work easier. On MultiWOZ 2.0 and MultiWOZ 2.1, the model achieved the best joint goal accuracy reported for the open vocabulary setting and ran much faster than TRADE. An error analysis shows that operation prediction is the main weak spot, which points to a clear direction for future work.

Key points

  • Joint goal accuracy is 51.72% on MultiWOZ 2.0 and 53.01% on MultiWOZ 2.1, rising to 53.68% with BERT-large. Code is released at github.com/clovaai/som-dst.
  • On the MultiWOZ 2.1 test set, SOM-DST generates values for 1.14 slots per turn on average and 9 at most. TRADE and ML-BST generate all 30 slots every turn, and COMER averages 5.72.
  • Latency on a Tesla V100 with batch size 1 is about 27 ms per turn, roughly 12.5 times faster than TRADE at 340 ms. This is close to NADST's 26 ms, while accuracy is clearly higher (53.01% vs 49.04%).
  • When the gold previous state is supplied, accuracy reaches 81.00%. Feeding back predicted states therefore increases the error about 2.47 times.
  • The state operation predictor accounts for roughly 78–93% of errors. Rare operations are handled poorly, with test F1 of 32.51 for DONTCARE and 2.86 for DELETE, which reflects strong class imbalance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.53

@inproceedings{kim2020efficient,
  title = {{Efficient Dialogue State Tracking by Selectively Overwriting Memory}},
  author = {Kim, Sungdong and Yang, Sohee and Kim, Gyuwan and Lee, Sang-Woo},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.53},
  url = {https://doi.org/10.18653/v1/2020.acl-main.53}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10–21, Berlin, Germany. Association for Computational Lingui… find our card
  2. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP. find
  3. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explorations Newsletter, 19(2):25–35. find
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014a. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In EMNLP. find
  5. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014b. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  7. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. find
  8. Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. In SIGDIAL. find our card
  9. Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. In Interspeech. find
  10. Hung Le, Doyen Sahoo, Chenghao Liu, Nancy F. Chen, and Steven C.H. Hoi. 2020a. End-to-end multi-domain task-oriented dialogue systems with multi-level neural belief tracker. In Submitted to ICLR 2020. find
  11. Hung Le, Richard Socher, and Steven C.H. Hoi. 2020b. Non-autoregressive dialog state tracking. In ICLR. find
  12. Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In ACL. find our card
  13. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… find our card
  14. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In ACL. find
  15. Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. In 2nd Conversational AI workshop on NeurIPS 2018. find
  16. Liliang Ren, Jianmo Ni, and Julian McAuley. 2019. Scalable and accurate dialogue state tracking via hierarchical sequence generation. In EMNLP-IJCNLP. find
  17. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  18. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958. find
  19. Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, et al. 2017. Nsml: A machine learning platform that enables you to focus on your models. arXiv preprint arXiv:1712.05902. find
  20. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In SIGDIAL. find
  21. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In EACL. find
  22. Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. 2018. Nsml: Meet the mlaas platform with a real-world case study. arXiv preprint arXiv:1810.09957. — [arXiv:1810.09957](https://arxiv.org/abs/1810.09957) find
  23. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In ACL. find
  24. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  25. Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In ACL. find
  26. Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Philip S. Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544. find our card
  27. Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2020-efficient-dialogue-state-tracking-selectively/