Elamur Research
☰
home / papers / 2019 / Dialogue Natural Language Inference

Dialogue Natural Language Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1363.pdf

Summary

Chit-chat agents often produce replies that clash with their earlier turns or with their assigned persona, for example by claiming two different jobs. Welleck et al. treat the detection of such clashes as natural language inference (NLI), where each sentence pair is classified as entailment, neutral, or contradiction. To get training data, they build Dialogue NLI from Persona-Chat. Crowdworkers on Amazon Mechanical Turk tag persona sentences and a subset of utterances with structured triples (entity, relation, entity). Pair labels are then derived automatically from these triples. Identical triples yield entailment, while opposing relations, swapped attribute values, or altered numbers yield contradictions. An NLI model trained on this data re-ranks the candidate replies of a persona-based Key-Value Memory Network and demotes candidates that conflict with the persona. On three purpose-built evaluation sets and in interactive human evaluation, re-ranking reduces persona contradictions and improves consistency. The work shows an NLI model delivering concrete gains on a downstream task, and it provides reusable automatic tests for dialogue consistency.

Key points

  • Dataset size and sources: Dialogue NLI has 310,110 training pairs, 16,500 validation pairs, 16,500 test pairs, and a 12,376-pair Test-Gold split. Pairs are (utterance, persona) and (persona, persona) sentences from Persona-Chat. The dataset is released at wellecks.github.io/dialogue_nli.
  • Label construction:
    • Neutral pairs come from miscellaneous utterances, persona pairing, and relation swaps between independent facts.
    • Contradictions come from relation swaps, entity swaps, and numeric replacements.
  • Evaluation resources: The authors introduce three evaluation sets (Haves, Likes, Attributes). They also define two new ranking metrics, Contradict@k and Entail@k, as variants of Hits@k.
  • Automatic results: With re-ranking (λ = 1.0, k = 10), Hits@1, Contradict@1, and Entail@1 all improve on every evaluation set.
  • Human evaluation: The study used ParlAI with 100 conversations per model, and scores were Bayesian-calibrated with Pyro and NUTS. KV-Mem + NLI outperformed KV-Mem on all calibrated scores:
    • overall score: 2.38 vs 2.21
    • share of consistent utterances: 0.35 vs 0.27
    • share of contradictions: 0.16 vs 0.25

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 132 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1363

@inproceedings{welleck2019dialogue,
  title = {{Dialogue Natural Language Inference}},
  author = {Welleck, Sean and Weston, Jason and Szlam, Arthur and Cho, Kyunghyun},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1363},
  url = {https://doi.org/10.18653/v1/p19-1363}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. 2018. Pyro: Deep Universal Probabilistic Programming. arXiv preprint arXiv:1810.09538. ↗
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, Christopher D Manning, and Stanford Linguistics. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. find
  3. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668. Association for Computational Linguistics. find
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loc Loc Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. pages 177–190. Springer, Berlin, Heidelberg. find
  6. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv preprint arXiv:1809.02922. ↗
  7. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. find
  8. Matthew D. Hoffman and Andrew Gelman. 2014. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593–1623. find
  9. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SCITAIL: A Textual Entailment Dataset from Science Question Answering. In AAAI. find
  10. Diederik P Kingma and Jimmy Lei Ba. 2014. Adam: A Method For Stochastic Optimization. arXiv preprint arXiv:1412.6980. ↗
  11. Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a Search Strategy in Neural Dialogue Modelling. arXiv preprint:1811.00907. ↗
  12. Wuwei Lan and Wei Xu. 2018. Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3890–3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  13. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A Persona-Based Neural Conversation Model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994–1003, Berlin, Germany. Association for Computational Linguistics. find our card
  14. Bill Maccartney and Christopher D Manning. 2009. An extended model of natural logic. Technical report. find
  15. M Marelli, S Menini, M Baroni, L Bentivogli, R Bernardi, and R Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), Reykjavik, Iceland. European Language Resources Association (ELRA). find
  16. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training Millions of Personalized Dialogue Agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics. find
  17. Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. ParlAI: A Dialog Research Software Platform. arXiv preprint:1705.06476. ↗ our card
  18. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  19. Adam Poliak, Yonatan Belinkov, James Glass, and Benjamin Van Durme. 2018a. On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 513–523, New Orleans, Louisiana. Association for Computational Linguistics. find
  20. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018b. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81. Association for Computational Linguistics. find our card
  21. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018c. Hypothesis Only Baselines in Natural Language Inference. In The Seventh Joint Conference on Lexical and Computational Semantics (*SEM). find
  22. Oriol Vinyals, Google Quoc, and V Le. 2015. A Neural Conversational Model. In ICML Deep Learning Workshop. find our card
  23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arXiv:1804.07461. ↗ our card
  24. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. find
  25. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing Dialogue Agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213, Melbourne, Australia. Association for Computational Linguistics. find our card