Elamur Research⌕
☰
home / papers / 2018 / Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing

Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing

Paper · 2018 · Osman Ramadan, Paweł Budzianowski and Milica Gašić

Summary

The paper tackles dialogue belief tracking when a conversation can move between several domains, such as hotels, restaurants, trains and taxis. Earlier trackers either depend on hand-built semantic dictionaries for delexicalisation or, like the Neural Belief Tracker (NBT), train each slot separately with parameters tied to the ontology. Both approaches scale poorly as domains are added. The authors propose a tracker that compares encoded user and system utterances with embeddings of ontology terms (domains, slots, values) in a shared semantic space. Because of this design, the parameter count does not grow with the ontology, and knowledge carries over between slots and domains. Domain detection and slot-value tracking are trained separately and then combined. A learned recurrent update across turns replaces hand-tuned update rules. The authors also release a new human-human Wizard-of-Oz corpus that is roughly ten times larger than earlier resources. The model beats NBT on single-domain tracking and handles multi-domain dialogues well, which makes it a practical basis for scalable task-oriented systems.

Key points

  • New corpus: It contains 2480 single-domain and 7375 multi-domain dialogues, each covering 1 to 5 domains. The combined ontology has 5 domains, 27 slots and 663 values. Test and development sets hold 1k dialogues each.
  • Architecture:
    • Encoders are either Bi-LSTMs or CNNs built on semantically specialised word embeddings, with 7 separate encoders in the Bi-LSTM variant.
    • Inform, request and confirm cases are modelled separately.
    • An RNN with a memory cell handles belief updates across turns.
  • WOZ 2.0 results (joint goal accuracy): NBT-CNN reaches 84.2, the Bi-LSTM variant 85.1 and the CNN variant 85.5.
  • New corpus, restaurant-only results (joint goal accuracy): NBT-CNN reaches 57.7, Bi-LSTM 59.9 and CNN 63.7. Scores are lower than on WOZ 2.0, which the authors attribute to noisier, richer dialogues.
  • Multi-domain test results: The Bi-LSTM variant scores F1 0.876 with 93.7% accuracy, and the CNN variant scores F1 0.878 with 93.2% accuracy. A uniform random baseline reaches only F1 0.108. The data and model are publicly available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 103 works

Cite

Cite as Osman Ramadan, Paweł Budzianowski and Milica Gašić. 2018. Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2069

@inproceedings{ramadan2018large,
  title = {{Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing}},
  author = {Osman Ramadan, Paweł Budzianowski and Milica Gašić},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2069},
  url = {https://doi.org/10.18653/v1/p18-2069}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5-6):602–610. find
  2. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings on EACL . find
  3. Matthew Henderson, Blaise Thomson, and Steve Young. 2014a. Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation. In Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, pages 360– 365. find
  4. John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the Association for Computational Linguistics 3:345–358.
  5. Jason Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse 7(3):4–33. find
  6. Matthew Henderson, Blaise Thomson, and Steve Young. 2014b. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). pages 292–299.
  7. Jason D Williams. 2014. Web-style ranking and slu combination for dialog state tracking. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). pages 282–291. find
  8. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of ACL .
  9. Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. 2017. Comparative study of cnn and rnn for natural language processing. arXiv preprint arXiv:1702.01923 . find
  10. John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS) 2(1):26–41. find
  11. Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thomson, and Kai Yu. 2010. The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language 24(2):150–174. find
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP .
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. ICLR . find
  14. Lukas Zilka and Filip Jurcicek. 2015. Incremental lstm-based dialog state tracker. In Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on. IEEE, pages 757–762. find
  15. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1777– 1788. find
  16. Nikola Mrkvsić and Ivan Vulić. 2018. Fully statistical neural belief tracking. In Proceedings of ACL. find
  17. Julien Perez and Fei Liu. 2016. Dialog state tracking, a machine reading approach using memory network. arXiv preprint arXiv:1606.04052 . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramadan-2018-large-scale-multi-domain-belief/