Elamur Research⌕
☰
home / papers / 2020 / Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking

Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking

Summary

Dialog state tracking (DST) in task-oriented assistants must infer, turn by turn, every <domain, slot, value> triplet a user wants. Earlier systems either pick values from a predefined ontology or copy spans from the conversation. Each approach fails in different cases. Full ontologies may be missing or very large, while span copying cannot recover values that users paraphrase or never state literally.

Zhang et al. propose DS-DST, which handles both cases with a single fine-tuned BERT reading-comprehension encoder. Each domain-slot name is concatenated with the dialog history so the two are encoded jointly. A slot gate handles special outcomes. Non-categorical slots, such as length of stay, are answered by predicting start and end positions. Categorical slots, such as parking yes/no, are answered by choosing from a partial picklist.

On MultiWOZ 2.0 and 2.1, the model clearly outperforms the BERT-DST baseline, which encodes slots and context separately. It sets a new state of the art on the noisier 2.0 and stays robust on the cleaner 2.1. The authors credit tight slot-context interaction for the gains and provide per-slot error analysis.

Key points

  • Three variants are compared:

    • DS-Span: all slots are treated as spans.
    • DS-DST: a hybrid of span and picklist slots.
    • DS-Picklist: all slots are picklist-based, which requires the full ontology.

  • Slot types are assigned using the heuristics from Rastogi et al. (2020). Slots with few possible values become categorical, and open-ended ones become non-categorical.

  • Encoder input format: [CLS] + domain-slot tokens + [SEP] + dialog context. The [CLS] vector and the token-level outputs feed the gate, the span predictor and the picklist scorer.

  • Error analysis shows recurring span-extraction failures:

    • implicit yes/no answers;
    • shortened mentions, such as "pool" for "swimming pool";
    • formatting mismatches, such as "8 : 15" vs. "08 : 15", that strict string matching penalizes.

  • The authors raise two concerns about MultiWOZ 2.0/2.1:

    • The datasets contain annotation noise.
    • Preprocessing differs across papers. Ignoring "none" values, for example, can inflate accuracy by up to 3%.

    They recommend MultiWOZ 2.2 and better-documented evaluation protocols.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

79 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhang2020find,
  title = {{Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking}},
  author = {Zhang, Jianguo and Hashimoto, Kazuma and Wu, Chien-Sheng and Wang, Yao and Yu, Philip and Socher, Richard and Xiong, Caiming},
  booktitle = {Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics},
  year = {2020},
  url = {https://aclanthology.org/2020.starsem-1.17/}
}

References

36 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2019. Learning to retrieve reasoning paths over wikipedia graph for question answering. arXiv preprint arXiv:1911.10470. find our card
  2. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP. find
  3. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. find
  4. Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al. 2019. The eighth dialog system technology challenge. arXiv preprint arXiv:1911.06394. — [arXiv:1911.06394](https://arxiv.org/abs/1911.06394) find
  5. Adarsh Kumar, Peter Ku, Anuj Kumar Goyal, Angeliki Metallinou, and Dilek Hakkani-Tur. 2020. Ma-dst: Multi-attention based scalable dialog state tracking. AAAI. find
  6. Hung Le, Richard Socher, and Steven CH Hoi. 2020. Non-autoregressive dialog state tracking. ICLR. find
  7. Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In ACL. find
  8. Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In AAAI, pages 7521–7528. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  10. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. find
  11. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In ACL. find
  12. Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In NIPS. find
  13. Bing Liu and Ian Lane. 2017. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. In INTERSPEECH. find
  14. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses: An Introduction. Wiley-Interscience. find
  15. Jianfeng Gao, Michel Galley, Lihong Li, et al. 2019a. Neural approaches to conversational ai. Foundations and Trends R in Information Retrieval, 13(2-3):127–298. find
  16. Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In ACL. find
  17. Shuyang Gao, Abhishek Sethi, Sanchit Aggarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019b. Dialog state tracking: A neural reading comprehension approach. In SIGDIAL. find our card
  18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  19. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014a. The second dialog state tracking challenge. In SIGDIAL. find
  20. Osman Ramadan, Paweł Budzianowski, and Milica Gašić. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In ACL. find
  21. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014b. The third dialog state tracking challenge. In IEEE SLT. find
  22. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. AAAI. find
  23. Matthew Henderson, Blaise Thomson, and Steve Young. 2014c. Word-based dialog state tracking with recurrent neural networks. In SIGDIAL. find our card
  24. Liliang Ren, Jianmo Ni, and Julian McAuley. 2019. Scalable and accurate dialogue state tracking via hierarchical sequence generation. EMNLP. find
  25. Seokhwan Kim, Luis Fernando D’Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 511–517. IEEE. find
  26. Yong Shan, Zekang Li, Jinchao Zhang, Fandong Meng, Yang Feng, Cheng Niu, and Jie Zhou. 2020. A contextual hierarchical attention network with adaptive objective for dialogue state tracking. arXiv preprint arXiv:2006.01554. find
  27. Seokhwan Kim, Luis Fernando D’Haro, Rafael E Banchs, Jason D Williams, and Matthew Henderson. 2017. The fourth dialog state tracking challenge. In Dialogues with Social Robots, pages 435– 449. Springer. find
  28. Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. In EACL. find
  29. Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In SIGDIAL. find
  30. Chien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming Xiong. 2020. Tod-bert: Pre-trained natural language understanding for task-oriented dialogues. arXiv preprint arXiv:2004.06871. — [arXiv:2004.06871](https://arxiv.org/abs/2004.06871) find
  31. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In ACL. find
  32. Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In ACL. find
  33. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. find
  34. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. arXiv preprint arXiv:2007.12720. find our card
  35. Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In ACL. find
  36. Li Zhou and Kevin Small. 2019. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192. — [arXiv:1911.06192](https://arxiv.org/abs/1911.06192) find

↑ back to top