Elamur Research⌕
☰
home / papers / 2019 / Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset

Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset

Paper · 2019 · Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara et al.

Summary

Virtual assistants such as Google Assistant or Alexa need to support an ever-growing number of services and APIs. The functionality of these services often overlaps, and many new services have almost no training data. Existing public corpora for task-oriented dialogue cover few domains and define a single fixed ontology for each domain. To address this, the authors from Google Research created the SGD corpus. Its dialogues are generated by a simulator: user and system agents interact with synthetic SQL-based services populated with real entities from Freebase. Crowdworkers then rewrite these structured outlines in natural language, and all annotations are preserved automatically. The authors also propose a schema-guided approach: a single model takes as input a service schema with natural-language descriptions of intents and slots, so a new API can be plugged in without additional training. Using BERT, they built a dialogue state tracking model that works in a zero-shot setting on unseen services and achieves competitive results on MultiWOZ. This work matters as a large benchmark resource for scalable dialogue systems.

Key points

  • The SGD training set contains 16,142 dialogues and 329,964 turns, with an average of 20.44 turns per dialogue. It covers 16 domains, 26 services, 214 slots, and 14,139 slot values. Across all three splits there are 20 domains and 45 services. The Alarm, Messaging, Payment, and Train domains appear only in dev/test, to evaluate generalization.
  • Slots are divided into categorical and non-categorical. Non-categorical slots (dates, movie titles) have no complete list of values. Each intent has required and optional slots, reflecting the constraints of real APIs.
  • On the SGD test set, the model trained on the full corpus (SGD-All) achieves a joint goal accuracy of 0.254 and an active intent accuracy of 0.906. On services not seen during training, results are noticeably worse, mainly due to the high proportion of OOV values. The weakest domains are RentalCars and Buses, with joint GA around 0.09.
  • On MultiWOZ 2.1, a variant of the model with pointer-generator-style copying reached a joint GA of 0.489, while the best known result at the time was 0.456.
  • The authors argue that simulation is more accurate and cheaper than Wizard-of-Oz: in MultiWOZ 2.0, annotation errors affected roughly 40% of turns. The corpus is published on GitHub as dstc8-schema-guided-dialogue.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Chatbots · 54 works

Cite

Cite as Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta and Pranav Khaitan. 2019. Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset. arXiv:1909.05855.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.05855 · arXiv 1909.05855

@misc{rastogi2019scalable,
  title = {{Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset}},
  author = {Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta and Pranav Khaitan},
  year = {2019},
  doi = {10.48550/arxiv.1909.05855},
  eprint = {1909.05855},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.05855}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bapna, A.; Tür, G.; Hakkani-Tür, D.; and Heck, L. P. 2017. Towards zero-shot frame semantic parsing for domain scaling. In Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017. find
  2. Budzianowski, P.; Wen, T.-H.; Tseng, B.-H.; Casanueva, I.; Ultes, S.; Ramadan, O.; and Gasic, M. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 5016–5026.
  3. Chao, G.-L., and Lane, I. 2019. BERT-DST: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. In INTERSPEECH. find
  4. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo…
  5. El Asri, L.; Schulz, H.; Sharma, S.; Zumer, J.; Harris, J.; Fine, E.; Mehrotra, R.; and Suleman, K. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 207– 219.
  6. Eric, M.; Goel, R.; Paul, S.; Sethi, A.; Agarwal, S.; Gao, S.; and Hakkani-Tur, D. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. Goel, R.; Paul, S.; and Hakkani-Tür, D. 2019. Hyst: A hybrid approach for flexible and accura… find
  7. Hemphill, C. T.; Godfrey, J. J.; and Doddington, G. R. 1990. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990. find
  8. Henderson, M.; Thomson, B.; and Williams, J. D. 2014a. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 263–272. find
  9. Henderson, M.; Thomson, B.; and Williams, J. D. 2014b. The third dialog state tracking challenge. 2014 IEEE Spoken Language Technology Workshop (SLT) 324–329. find
  10. Henderson, M.; Thomson, B.; and Young, S. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 292–299.
  11. Hendrycks, D., and Gimpel, K. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. find
  12. Kelley, J. F. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS) 2(1):26–41. find
  13. Kim, S.; DHaro, L. F.; Banchs, R. E.; Williams, J. D.; and Henderson, M. 2017. The fourth dialog state tracking challenge. In Dialogues with Social Robots. Springer. 435–449. find
  14. Kim, Y.-B.; Kim, D.; Kumar, A.; and Sarikaya, R. 2018. Efficient large-scale neural domain classification with personalized attention. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2214–2224. Melbourne, Australia: Association for… find
  15. Mrkšić, N.; Séaghdha, D. Ó.; Wen, T.-H.; Thomson, B.; and Young, S. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, 1777–1788. find
  16. Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
  17. Rastogi, A.; Gupta, R.; and Hakkani-Tur, D. 2018. Multi-task learning for joint language understanding and dialogue state tracking. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, 376–384. Rastogi, A.; Hakkani-Tür, D.; and Heck, L. 2017. Scalable multi-domain dialogue s… find
  18. Ren, L.; Xie, K.; Chen, L.; and Yu, K. 2018. Towards universal dialogue state tracking. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2780–2786. Shah, P.; Hakkani-Tür, D.; Tür, G.; Rastogi, A.; Bapna, A.; Nayak, N.; and Heck, L. 2018. Building a convers…
  19. Shah, D.; Gupta, R.; Fayazi, A.; and Hakkani-Tur, D. 2019. Robust zero-shot cross-domain slot filling with example values. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5484–5490. Florence, Italy: Association for Computational Linguistics. Wen, T.; Vandy… find
  20. Williams, J.; Raux, A.; Ramachandran, D.; and Black, A. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, 404–413. find
  21. Wu, C.-S.; Madotto, A.; Hosseini-Asl, E.; Xiong, C.; Socher, R.; and Fung, P. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 808– 819. Florence, Italy: Association for Com…
  22. Xia, C.; Zhang, C.; Yan, X.; Chang, Y.; and Yu, P. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3090–3099. Association for Computational Linguistics.
  23. Yang, Z.; Salakhutdinov, R.; and Cohen, W. W. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint arXiv:1703.06345. find
  24. Zhong, V.; Xiong, C.; and Socher, R. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1458–1467. Melbourne, Australia: Association for Computational Linguistics… find
  25. Zhu, S., and Yu, K. 2018. Concept transfer learning for adaptive language understanding. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, 391–399. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rastogi-2019-towards-scalable-multi-domain-conversational/