Elamur Research⌕
☰
home / papers / 2020 / Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

Paper · 2020 · Margaret Li, Stephen Roller, Ilia Kulikov et al.

Summary

Generative dialogue models trained only with maximum likelihood show several weaknesses. They copy too much from the context, repeat n-grams inside a single reply, overuse frequent tokens, and can contradict what was said earlier. The authors extend the unlikelihood loss of Welleck et al. (2019a) into a general framework. Alongside standard likelihood, it adds a term that lowers the probability of chosen "negative candidate" tokens. Different choices of candidates and per-token weights target different flaws:

  • n-grams copied from the context or repeated within the generated reply,
  • tokens the model produces more often than humans, weighted by the gap between the model's and the human unigram distributions,
  • whole contradictory responses built from Dialogue NLI labels.

All experiments fine-tune a seq2seq Transformer pre-trained on pushshift.io Reddit. The statistical penalties move generated text closer to human statistics. The NLI-based training teaches the model to assign much lower probability to contradictions, while overall perplexity on ConvAI2 barely changes. This matters because it shows that labeled data about unwanted behavior can be built directly into generator training, rather than only used to rescore outputs.

Key points

  • Base model: a Transformer with an 8-layer encoder and an 8-layer decoder, 512-dimensional embeddings and 16 attention heads, built in ParlAI. It was pre-trained on roughly 2200M pushshift.io Reddit examples.
  • The MLE baseline reaches a perplexity of 11.4 on ConvAI2, and unlikelihood training raises it only slightly, to about 11.8–11.9.
  • On the Dialogue NLI two-utterance task, the baseline assigns lower perplexity to contradictions (12.5) than to neutral (36.7) or triple-entailing (17.5) replies. After unlikelihood training, contradiction perplexity rises to 248.9, compared with 9.1 for entailing replies.
  • On the same task, selection accuracy for neutral over contradicting replies grows from 18% to 78%, and for entailing replies from 72% to 96%.
  • On the full-dialogue variant, neutral selection accuracy rises to 69.8% from about 37%. Code and models are released via parl.ai (dialogue_unlikelihood project).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🎓New York UniversityNew York · United States136 works · 86,390 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Dialogue systems · 401 works

Cite

Cite as Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho and Jason Weston. 2020. Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.428

@inproceedings{li2020don,
  title = {{Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training}},
  author = {Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho and Jason Weston},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.428},
  url = {https://doi.org/10.18653/v1/2020.acl-main.428}
}

References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  2. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The second conversational… our card
  3. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations. find
  4. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54. Association for Computational Linguistics. our card
  5. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. our card
  6. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94–104, Copenhagen, Denmark. Association for Computational Linguistics. our card
  7. Saadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli Celikyilmaz, and Yejin Choi. 2019. Cooperative generator-discriminator networks for abstractive summarization with narrative flow. arXiv preprint arXiv:1907.01272. — [arXiv:1907.01272](https://arxiv.org/abs/1907.01272) find
  8. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pages 43–48, Vancouver, Canada. Association for Computational Linguistics. our card
  9. Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, and Graham Neubig. 2019. Latent relation language models. arXiv preprint arXiv:1908.07690. — [arXiv:1908.07690](https://arxiv.org/abs/1908.07690) find
  10. Tianxing He and James Glass. 2019. Negative training for neural dialogue response generation. arXiv preprint arXiv:1903.02134. find
  11. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. find
  12. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969. find
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. our card
  14. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. In Proceedings of the NeurIPS Workshop on Conversational AI. find
  15. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen… our card
  16. Aigul Nugmanova, Andrei Smirnov, Galina Lavren-tyeva, and Irina Chernykh. 2019. Strategy of the negative sampling for training retrieval-based dialogue systems. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 844– 848. IEEE. find
  17. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. our card
  18. Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  20. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… our card
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. our card
  22. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019a. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319. our card
  23. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019b. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. our card
  24. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  25. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 164– 174, Melbourne,… find
  26. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  27. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213, Melbo… our card
  28. Token frequency classes Model PPL F1 Context Label — .400 .300 11.4 .199 .491 .282 F1 Freq Med Rare Rarest .200 .157 .100 .068 Human MLE Baseline — 8.3 .373 .160 .0006 .582 .002 UL (Context only) UL (Label only) UL (Context + Label) 8.8 .345 8.3 .371 8.5 .358 .270 .645 .445 Nucleus p = 0.3 Nucleus… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-don-t-say-that-making/