Elamur Research⌕
☰
home / papers / 2017 / Adversarial Learning for Neural Dialogue Generation

Adversarial Learning for Neural Dialogue Generation

Summary

Neural chatbots trained with maximum likelihood tend to produce bland, repetitive, short-sighted replies. Hand-crafted RL rewards only capture a few chosen qualities. The authors instead take inspiration from the Turing test. A SEQ2SEQ generator is trained jointly with a discriminator, a hierarchical LSTM classifier that tries to tell human responses from machine ones given the dialogue context. The discriminator's estimated probability that a reply is human-written becomes the generator's reward, optimized with REINFORCE plus a learned critic baseline.

To give each token its own credit, they propose REGS (reward for every generation step). It works either through Monte Carlo rollouts or through a discriminator trained on partial sequences.

They also design adversarial evaluation (AdverSuc), which measures how often a model fools a trained evaluator. It is paired with a sanity check against random responses.

The adversarial systems beat MLE and mutual-information baselines on AdverSuc. In pairwise human judgments they improve both single-turn and multi-turn quality. This suggests a learned "human-likeness" reward is a more general training signal than manually defined ones.

Key points

  • Architecture:
    • The generator is an encoder-decoder RNN.
    • The discriminator is a hierarchical LSTM over the context plus the response, ending in a 2-class softmax.
    • The context is approximated by the two preceding utterances.
  • REGS with Monte Carlo search: each partial sequence is completed N = 5 times by sampling, and the discriminator's scores on these completions are averaged.
  • AdverSuc results: Adver-REGS scored 0.098 and Adver-Reinforce 0.080, versus 0.073 for MMI+p(t|s), 0.049 for MLE-greedy and 0.037 for MLE-BS.
    • Sampling reached 0.372, but its machine-vs-random accuracy dropped to 0.679, versus about 0.94–0.95 for the other models.
    • This means the evaluator could not separate sampled replies even from random ones, so the high score is misleading.
  • Human evaluation against the mutual-information system: 200 items, 3 crowdsourced judges per item.
    • Single-turn: the adversarial model won 62%, lost 18%, tied 20%.
    • Multi-turn (3-turn simulated conversations): won 72%, lost 10%, tied 18%.
  • Machine translation: preliminary MT experiments showed no clear gain. The authors conjecture that adversarial training helps most when target outputs have high entropy.

Interesting ideas

  • I1. Learned reward instead of hand-written criteria: a classifier that separates real from generated outputs can serve as the reward. This avoids enumerating desirable properties by hand and could transfer to other open-ended generation tasks where "good" is hard to specify.
  • I2. Per-token credit assignment: scoring partial outputs, by rolling them out with Monte Carlo sampling or by training the discriminator on prefixes, gives finer feedback than one sequence-level reward. This trick fits any RL-trained sequence model.
  • I3. Sanity check for evaluators: fooling a classifier is only meaningful if that classifier can still tell the model's outputs from random responses. Reporting machine-vs-random accuracy next to the main score exposes degenerate "wins" such as noisy sampling. This is a useful safeguard for any learned metric.
  • I4. Learned critic as baseline: a separate network predicts the expected reward from the dialogue history and is subtracted from the actual reward. This reduces gradient variance without biasing the estimate, a standard but effective stabilizer for policy-gradient training on text.
  • I5. Predicting where adversarial training pays off: the authors link the method's benefit to output entropy, with dialogue gaining and MT not. This gives a quick heuristic for choosing tasks before investing effort.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

171 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 154 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1230

@inproceedings{li2017adversarial,
  title = {{Adversarial Learning for Neural Dialogue Generation}},
  author = {Li, Jiwei and Monroe, Will and Shi, Tianlin and Jean, Sébastien and Ritter, Alan and Jurafsky, Dan},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1230},
  url = {https://doi.org/10.18653/v1/d17-1230}
}

References

50 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. V. M. Aleksandrov, V. I. Sysoyev, and V. V. Shemeneva. 1968. Stochastic optimization. Engineering Cybernetics 5:11–16. find
  2. Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems. pages 4601–4609. find
  3. Nabiha Asghar, Pasca Poupart, Jiang Xin, and Hang Li. 2016. Online sequence-to-sequence reinforcement learning for open-domain conversational agents. arXiv preprint arXiv:1612.03929 . — [arXiv:1612.03929](https://arxiv.org/abs/1612.03929) find
  4. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proc. of NAACL-HLT. find our card
  5. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. ICLR . find our card
  6. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 994–1003. http://… find our card
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  8. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. CoNLL . find our card
  9. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016a. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances In Neural Information Processing Systems. pages 2172–2180. find
  10. Xilun Chen, Ben Athiwaratkun, Yu Sun, Kilian Weinberger, and Claire Cardie. 2016b. Adversarial deep averaging networks for cross-lingual sentiment classification. arXiv preprint arXiv:1606.01614 . find our card
  11. Emily L Denton, Soumith Chintala, Rob Fergus, et al. 2015. Deep generative image models using a? laplacian pyramid of adversarial networks. In Advances in neural information processing systems. pages 1486–1494. find
  12. Peter W Glynn. 1990. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM 33(10):75–84. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  14. Thorsten Joachims. 2002. Learning to classify text using support vector machines: Methods, theory and algorithms. Kluwer Academic Publishers. find
  15. Anjuli Kannan and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training. find
  16. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. ACL . find
  17. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016c. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562 . — [arXiv:1611.08562](https://arxiv.org/abs/1611.08562) find
  18. Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016d. Deep reinforcement learning for dialogue generation. EMNLP . find our card
  19. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. EMNLP . find
  20. Ryan Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. ACL . find our card
  21. Ryan Lowe, Iulian V Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. SIGDIAL . find
  22. Yi Luan, Yangfeng Ji, and Mari Ostendorf. 2016. LSTM based conversation models. arXiv preprint arXiv:1603.09457 . — [arXiv:1603.09457](https://arxiv.org/abs/1603.09457) find
  23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. ACL . find our card
  24. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. ACL . find
  25. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489. find
  26. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 . find our card
  27. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. ICLR . find our card
  28. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of EMNLP 2011. pages 583–593. find our card
  29. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. pages 2226–2234. find our card
  30. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of AAAI. find our card
  31. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016b. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). find our card
  32. Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2016c. Multiresolution recurrent neural networks: An application to dialogue response generation. arXiv preprint arXiv:1606.00776 . — [arXiv:1606.00776](https://arxiv.org/abs/1606.00776) find
  33. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016d. Generative deep neural networks for dialogue: A short review . find
  34. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. AAAI . find our card
  35. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of NAACL-HLT. find our card
  36. Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arxiv . find
  37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
  38. Alan M Turing. 1950. Computing machinery and intelligence. Mind 59(236):433–460. find
  39. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of ICML Deep Learning Workshop. find our card
  40. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562 . find
  41. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256. find
  42. Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. ACL . find
  43. Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2016. Incorporating loose-structured knowledge into LSTM with recall gate for conversation modeling. arXiv preprint arXiv:1605.05110 . — [arXiv:1605.05110](https://arxiv.org/abs/1605.05110) find
  44. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of ACL-IJCNLP. pages 1577–1586. find our card
  45. Kaisheng Yao, Geoffrey Zweig, and Baolin Peng. 2015. Attention with intention for a neural network conversation model. In NIPS workshop on Machine Learning for Spoken Language Understanding and Interaction. find
  46. Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating long and diverse responses with neural conversational models. ICLR . find
  47. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2016a. Seqgan: sequence generative adversarial nets with policy gradient. arXiv preprint arXiv:1609.05473 . — [arXiv:1609.05473](https://arxiv.org/abs/1609.05473) find
  48. Zhou Yu, Ziyu Xu, Alan W Black, and Alex I Rudnicky. 2016b. Strategy and policy learning for non-task-oriented conversational systems. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. page 404. find
  49. Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2017. Aspect-augmented adversarial networks for domain adaptation. arXiv preprint arXiv:1701.00188 . find
  50. Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanill… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/