Elamur Research⌕
☰
home / papers / 2019 / Dialog State Tracking: A Neural Reading Comprehension Approach

Dialog State Tracking: A Neural Reading Comprehension Approach

Summary

Dialog state tracking (DST) records the slot values a user expresses during a task-oriented conversation. Most earlier trackers pick values from a fixed ontology, so they fail on values unseen in training and scale poorly to large vocabularies. Researchers at Amazon Alexa AI recast DST as machine reading comprehension. The dialog history acts as the passage, and each slot becomes a question asking for its value.

The tracker makes three decisions in order:

  • A slot carryover model decides whether to keep the previous turn's value.
  • A slot type model chooses among Yes, No, DontCare or Span.
  • If a span is needed, an attention-based slot span model locates its start and end tokens in the dialog.

The dialog is encoded with BERT embeddings and a bidirectional LSTM. On MultiWOZ-2.0, this simple pointer approach beats more elaborate earlier systems. Combining it with a closed-vocabulary tracker improves results further. The work shows that span extraction can handle open-ended slot values, and it identifies carryover as the main bottleneck.

Key points

  • On MultiWOZ-2.0, the span-based model reaches 42.12% joint-goal accuracy. This is about 6.5 points above the previous best result.
  • Combining it with the fixed-vocabulary joint tracker of Liu and Lane (2017) raises joint-goal accuracy to 47.33%, a gain of 11.75 points. The authors note that Wu et al. (2019) later surpassed this.
  • In the authors' error analysis, "Unanswerable Slot" cases account for 65.5% of errors and imprecise slot resolution for 12.9%. Oracle ablations also show that a perfect carryover model would help most.
  • Per-turn carryover accuracy stays near 75% for every context encoder tested: flat LSTM, hierarchical, and self-attention. The hierarchical encoder over all turns scores best at 75.98%.
  • Errors accumulate with conversation depth. The share of incorrect turns rises from 23.90% at turn 1 to 94.77% at turn 10.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 63 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5932

@inproceedings{gao2019dialog,
  title = {{Dialog State Tracking: A Neural Reading Comprehension Approach}},
  author = {Gao, Shuyang and Sethi, Abhishek and Agarwal, Sanchit and Chung, Tagyoung and Hakkani-Tur, Dilek},
  booktitle = {Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue},
  year = {2019},
  doi = {10.18653/v1/w19-5932},
  url = {https://doi.org/10.18653/v1/w19-5932}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julien Perez and Fei Liu. 2017. Dialog state tracking, a machine reading approach using memory network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 305–314. find
  2. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… find
  3. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  4. Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University. find
  5. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL). find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. Osman Ramadan, Paweł Budzianowski, and Milica Gasic. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 432–437. find
  9. Rahul Goel, Shachi Paul, Tagyoung Chung, Jeremie Lecomte, Arindam Mandal, and Dilek Hakkani-Tur. 2018. Flexible and scalable state tracking framework for goal-oriented dialogue systems. arXiv preprint arXiv:1811.12891. — [arXiv:1811.12891](https://arxiv.org/abs/1811.12891) find
  10. Abhinav Rastogi, Dilek Hakkani-Tür, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. In Automatic Speech Recognition and Understanding Workshop (ASRU), 2017 IEEE, pages 561– 568. IEEE. find
  11. Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. find
  12. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association of Computational Linguistics (TACL). find our card
  13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693– 1701. find our card
  14. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203. find our card
  15. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317–328. find our card
  16. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. find our card
  17. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  18. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  20. Bing Liu and Ian Lane. 2017. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. find
  21. Proc. Interspeech 2017, pages 2506–2510. find
  22. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855. find our card
  23. Jason D Williams, Pascal Poupart, and Steve Young. 2005. Factored partially observable markov decision processes for dialogue management. In Proc. IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 76–82. find
  24. Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), 2nd Conversational AI workshop. find
  25. Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393–422. find
  26. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819. find
  27. Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). find
  28. Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive dialogue state tracker. arXiv preprint arXiv:1805.09655. — [arXiv:1805.09655](https://arxiv.org/abs/1805.09655) find
  29. Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593. — [arXiv:1812.03593](https://arxiv.org/abs/1812.03593) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gao-2019-dialog-state-tracking-neural-reading/