Elamur Research⌕
☰
home / papers / 2020 / End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2

End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2

Paper · 2020 · Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang et al.

Summary

Goal-oriented dialogue systems are usually built in one of two ways, and each has a drawback. Modular pipelines (NLU, DST, POL, NLG) tune each component separately, so the whole system may not improve. Monolithic end-to-end models learn only from utterance pairs, which ignores corpus annotations and makes database access and explanation difficult. Ham et al. fine-tune a single GPT-2 model on delexicalized MultiWOZ dialogues so that it reproduces the pipeline in sequence. Given the dialogue history, it first predicts the dialogue state and then the system action. A separate module then queries the database, with a fallback when the query returns nothing. Finally, the model generates the response. Training uses ordinary gradient descent. Because the state and action are produced as intermediate outputs, the model is easier to interpret and to connect to external systems. Evaluated through ConvLab, the system took first place in the human evaluation of the DSTC8 end-to-end multi-domain task. This shows that pretrained language models can yield natural, task-completing dialogue without losing the structure of a pipeline.

Key points

  • In DSTC8 human evaluation, the system with top-p sampling (p=0.8) achieved a 68.32% success rate, a Language Understanding score of 4.149 and a Response Appropriateness score of 4.287. It ranked 1st, beating the runner-up by 2.51% success, 0.365 points and 0.458 points respectively.
  • In automatic evaluation with the ConvLab user simulator (500 dialogues, greedy decoding), it reached a 78.60% success rate, an 86.34% book rate and an F1 of 0.87. This beat the baseline but not every submitted system, which the authors attribute mainly to intent-recognition errors in the simulator.
  • The model is GPT2-small (124M parameters, 12 decoder blocks). It was fine-tuned for 4 epochs with batch size 2, Adam (lr 6.25e-5), a history limit of 15 and LM/NC loss weights of 2.0/1.0.
  • On the MultiWOZ Dialogue State Tracking and Dialogue-Context-to-Text Generation benchmarks, it is competitive with specialized models without task-specific tuning. Attention visualizations show it attending to the tokens produced at earlier pipeline steps.
  • Code is released at github.com/KAIST-AILab/NeuralPipeline_DSTC8.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea17 works · 2,673 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksDialogue systems · 401 works

Cite

Cite as Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang and Kee-Eung Kim. 2020. End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.54

@inproceedings{ham2020end,
  title = {{End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2}},
  author = {Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang and Kee-Eung Kim},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.54},
  url = {https://doi.org/10.18653/v1/2020.acl-main.54}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research.
  2. Paweł Budzianowski and Ivan Vulić. 2019. Hello, It’s GPT-2 – How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems. In Proceedings of the 3rd Workshop on Neural Generation and Translation. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language… find
  4. Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  5. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2018. The Second Conversational Intelligence Challenge (ConvAI2). In The NeurIPS’18 Competition.
  6. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  7. Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, Minlie Huang, Luis Lastras, Jonathan K. Kummerfeld, Walter S. Lasecki, Chiori Hori, Anoop Cherian, Tim K. Marks, Abhinav Rastogi, Xiaoxue Zang, Srinivas S… find
  8. Young-Bum Kim, Sungjin Lee, and Karl Stratos. 2017. OneNet: Joint Domain, Intent, Slot Prediction for Spoken Language Understanding. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop. find
  9. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations. find
  10. Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019a. SUMBT: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
  11. Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019b. ConvLab: Multi-Domain End-to-End Dialog System Platform. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  12. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
  13. Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
  14. Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. 2019. Structured fusion networks for dialog. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  15. Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking. In NeurIPS 2018, 2nd Conversational AI workshop. find
  16. Jiahuan Pei, Pengjie Ren, and Maarten de Rijke. 2019. A modular task-oriented dialogue system using a neural mixture-of-experts. In WCIS: SIGIR 2019 Workshop on Conversational Interaction Systems. find
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language understanding paper.pdf. ↗
  18. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multi-task learners. https://www.techbooky.com/wp-content/uploads/2019/02/Better-Language-Models-and-Their-Implications.pdf. ↗
  19. Osman Ramadan, Paweł Budzianowski, and Milica Gašić. 2018. Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  21. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  22. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computati… find
  23. Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, Brussels, Belgium. Association for Computational Linguist…
  24. Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The Dialog State Tracking Challenge. In Proceedings of the SIGDIAL 2013 Conference. find
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint abs:1910.03771.
  26. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2018. TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents. find
  27. NeurIPS 2018 workshop on Conversational AI: “Today’s Practice and Tomorrow’s Potential”. find
  28. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  29. Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech…
  30. Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ham-2020-end-end-neural-pipeline-goal/