Elamur Research⌕
☰
home / papers / 2020 / A Simple Language Model for Task-Oriented Dialogue

A Simple Language Model for Task-Oriented Dialogue

Paper · 2020 · Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu et al.

Summary

Dialogue systems that help a user accomplish a specific task (task-oriented dialogue, TOD) are usually built as a pipeline of separate modules. One module tracks the dialogue state (belief state), a second selects system actions, and a third generates the response. Because of this division, errors from one module propagate to the subsequent ones. Researchers from Salesforce Research propose SimpleTOD, a single causal language model that solves all three subtasks as a single sequence prediction task. The model reads the dialogue history and generates the belief state. It then receives aggregated results of the database query and generates the actions and a delexicalized response. The weights are taken from GPT-2 or DistilGPT2, and the model is fine-tuned on MultiWOZ 2.1. Greedy decoding is sufficient for generation. SimpleTOD outperformed previous methods both in dialogue state tracking and in end-to-end evaluation of actions and responses. Thus, the knowledge the model acquired during pre-training on general text transfers well to target dialogues, and a cumbersome modular architecture can be replaced with a simple one. The authors released the code, a list of erroneous annotations in MultiWOZ, and a cleaned version of the test set.

Key points

  • Joint goal accuracy on MultiWOZ 2.1 is 55.76, the best result for state tracking at the time of publication. In the end-to-end setting, the inform rate increased by 8.1 points, the success rate by 9.7, and the combined score by 7.2.
  • Special tokens are critically important. Without end-of-segment markers and without user/system turn tagging, joint accuracy drops to 16.79, and the combined score falls from 92.98 to 26.73.
  • Without pre-training, the model performs much worse: a 12-layer model with random weights achieves a joint accuracy of 20.17 versus 55.76 for GPT-2. A larger model is not always better: DistilGPT2 yields a combined score of 92.98, while GPT-2 yields 90.75.
  • The authors identified four types of erroneous annotations in MultiWOZ 2.1 and showed that SimpleTOD is robust to such noise.
  • The code, the cleaned test set, and the list of annotation errors were released on GitHub: salesforce/simpletod. The paper was published at NeurIPS 2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States56 works · 17,867 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDialogue systems · 624 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Chatbots · 54 works

Cite

Cite as Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz and Richard Socher. 2020. A Simple Language Model for Task-Oriented Dialogue. arXiv:2005.00796.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.00796 · arXiv 2005.00796

@misc{hosseiniasl2020simple,
  title = {{A Simple Language Model for Task-Oriented Dialogue}},
  author = {Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz and Richard Socher},
  year = {2020},
  doi = {10.48550/arxiv.2005.00796},
  eprint = {2005.00796},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.00796}
}

References

64 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kulshreshtha, G. Nemade, Y. Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020.
  2. J. Ba, R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016.
  3. S. Bao, H. He, F. Wang, and H. Wu. Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931, 2019.
  4. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003.
  5. A. Bordes, Y.-L. Boureau, and J. Weston. Learning end-to-end goal-oriented dialog. In International Conference on Learning Representations, 2017. find
  6. P. Budzianowski and I. Vulić. Hello, it’s gpt-2–how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774, 2019. find
  7. P. Budzianowski, I. Casanueva, B.-H. Tseng, and M. Gasic. Towards end-to-end multi-domain dialogue modelling. 2018. find
  8. L. Chen, B. Lv, C. Wang, S. Zhu, B. Tan, and K. Yu. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. 2020. find
  9. W. Chen, J. Chen, P. Qin, X. Yan, and W. Y. Wang. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. arXiv preprint arXiv:1905.12866, 2019. find
  10. R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  12. L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13042–13054, 2019.
  13. M. Eric and C. D. Manning. Key-value retrieval networks for task-oriented dialogue. arXiv preprint arXiv:1705.05414, 2017.
  14. M. Eric, R. Goel, S. Paul, A. Sethi, S. Agarwal, S. Gao, and D. Hakkani-Tur. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669, 2019. find
  15. J. Gao, M. Galley, L. Li, et al. Neural approaches to conversational ai. Foundations and Trends® in Information Retrieval, 13(2-3):127–298, 2019. find
  16. S. Gao, Y. Zhang, Z. Ou, and Z. Yu. Paraphrase augmented task-oriented dialog generation. arXiv preprint arXiv:2004.07462, 2020. find
  17. D. Ham, J.-G. Lee, Y. Jang, and K.-E. Kim. End-to-end neural pipeline for goal-oriented dialogue system using gpt-2. ACL, 2020. find
  18. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  19. M. Heck, C. van Niekerk, N. Lubis, C. Geishauser, H.-C. Lin, M. Moresi, and M. Gašić. Trippy: A triple copy strategy for value independent neural dialog state tracking. arXiv preprint arXiv:2005.02877, 2020. find
  20. M. Henderson, B. Thomson, and S. Young. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, 2013. find
  21. M. Henderson, I. Casanueva, N. Mrkšić, P.-H. Su, I. Vulić, et al. Convert: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688, 2019.
  22. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  23. N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
  24. W. Lei, X. Jin, M.-Y. Kan, Z. Ren, X. He, and D. Yin. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018.
  25. W. Liang, Y. Tian, C. Chen, and Z. Yu. Moss: End-to-end dialog system framework with modular supervision. arXiv preprint arXiv:1909.05528, 2019. — [arXiv:1909.05528](https://arxiv.org/abs/1909.05528) find
  26. B. Liu and I. Lane. Attention-based recurrent neural network models for joint intent detection and slot filling. In INTERSPEECH, 2016. find
  27. B. Liu and I. Lane. End-to-end learning of task-oriented dialogs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 67–73, 2018. find
  28. B. Liu, G. Tür, D. Hakkani-Tür, P. Shah, and L. Heck. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  29. A. Madotto, C.-S. Wu, and P. Fung. Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. arXiv preprint arXiv:1804.08217, 2018.
  30. B. McCann, J. Bradbury, C. Xiong, and R. Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305, 2017.
  31. S. Mehri, T. Srinivasan, and M. Eskenazi. Structured fusion networks for dialog. arXiv preprint arXiv:1907.10016, 2019. find
  32. N. Mrkšić, D. Ó Séaghdha, T.-H. Wen, B. Thomson, and S. Young. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017. find
  33. A. Neelakantan, S. Yavuz, S. Narang, V. Prasad, B. Goodrich, D. Duckworth, C. Sankar, and X. Yan. Neural assistant: Joint action prediction, response generation, and latent knowledge reasoning. In NeurIPS 2019 Converstional AI Workshop, 2019. find
  34. E. Nouri and E. Hosseini-Asl. Toward scalable neural dialogue state tracking model. In NeurIPS 2018 Conversational AI Workshop, 2018. find
  35. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: A method for automatic evaluation of machine translation. In ACL, 2002.
  36. B. Peng, C. Li, J. Li, S. Shayandeh, L. Liden, and J. Gao. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. arXiv preprint arXiv:2005.05298, 2020. find
  37. B. Peng, C. Zhu, C. Li, X. Li, J. Li, M. Zeng, and J. Gao. Few-shot natural language generation for task-oriented dialog, 2020.
  38. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
  39. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. find
  40. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
  41. A. Rastogi, D. Hakkani-Tur, and L. Heck. Scalable multi-domain dialogue state tracking. In Proceedings of IEEE ASRU, 2017. find
  42. S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637, 2020.
  43. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. find
  44. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistic…
  45. L. Shu, P. Molino, M. Namazifar, B. Liu, H. Xu, H. Zheng, , and G. Tur. Incorporating the structure of the belief state in end-to-end task-oriented dialogue systems. In NeurIPS 2018 Converstional AI Workshop, 2018. find
  46. R. W. Smith and D. R. Hipp. Spoken natural language dialog systems: A practical approach. Oxford University Press on Demand, 1994. find
  47. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
  48. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
  49. T.-H. Wen, M. Gašić, N. Mrkšić, P.-H. Su, D. Vandyke, and S. Young. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.
  50. T.-H. Wen, D. Vandyke, N. Mrksic, M. Gasic, L. M. Rojas-Barahona, P.-H. Su, S. Ultes, and S. Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562, 2016.
  51. T.-H. Wen, Y. Miao, P. Blunsom, and S. Young. Latent intention dialogue models. In Proceedings of the 34th International Conference on Machine Learning, 2017. find
  52. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  53. C.-S. Wu, A. Madotto, E. Hosseini-Asl, C. Xiong, R. Socher, and P. Fung. Transferable multi-domain state generator for task-oriented dialogue systems. arXiv preprint arXiv:1905.08743, 2019.
  54. C.-S. Wu, R. Socher, and C. Xiong. Global-to-local memory pointer networks for task-oriented dialogue. arXiv preprint arXiv:1901.04713, 2019. find
  55. C.-S. Wu, S. Hoi, R. Socher, and C. Xiong. Tod-bert: Pre-trained natural language understanding for task-oriented dialogues. arXiv preprint arXiv:2004.06871, 2020. — [arXiv:2004.06871](https://arxiv.org/abs/2004.06871) find
  56. Q. Wu, Y. Zhang, Y. Li, and Z. Yu. Alternating recurrent dialog model with large-scale pre-trained language models. arXiv preprint arXiv:1910.03756, 2019. find
  57. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764, 2019.
  58. S. Young, M. Gašić, B. Thomson, and J. D. Williams. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179, 2013. find
  59. J.-G. Zhang, K. Hashimoto, C.-S. Wu, Y. Wan, P. S. Yu, R. Socher, and C. Xiong. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544, 2019.
  60. Y. Zhang, Z. Ou, and Z. Yu. Task-oriented dialog systems that consider multiple appropriate responses under the same context. arXiv preprint arXiv:1911.10484, 2019. — [arXiv:1911.10484](https://arxiv.org/abs/1911.10484) find
  61. Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536, 2019.
  62. T. Zhao, A. Lu, K. Lee, and M. Eskenazi. Generative encoder-decoder models for task-oriented spoken dialog systems with chatting capability. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 2017. find
  63. T. Zhao, K. Xie, and M. Eskenazi. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. arXiv preprint arXiv:1902.08858, 2019.
  64. L. Zhou and K. Small. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192, 2019. — [arXiv:1911.06192](https://arxiv.org/abs/1911.06192) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hosseini-asl-2020-simple-language-model-task-oriented/