Elamur Research
☰
home / papers / 2020 / Uncertain Natural Language Inference

Uncertain Natural Language Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.774.pdf

Summary

Standard NLI treats entailment as a choice among three discrete labels, which hides how much pairs with the same label can differ in plausibility. The authors propose Uncertain Natural Language Inference (UNLI), in which a system predicts a human's subjective probability that a hypothesis holds given a premise. To build data, they had MTurk workers re-annotate part of SNLI using an EASL-style interface. The slider was stretched logistically so that values close to 0 and 1 could be told apart, since people are especially sensitive to differences at those extremes. The resulting corpus, u-SNLI, shows that entailed, neutral and contradicted pairs each cover a broad range of probabilities. For modeling, they adapted BERT into a regressor with a sigmoid output trained with binary cross-entropy. Pre-training on SNLI with its categorical labels mapped to scalars, followed by fine-tuning on u-SNLI, worked best and came close to human agreement. Errors point to weak commonsense and plausibility reasoning. The authors argue that future NLI datasets should use scalar annotation.

Key points

  • u-SNLI corpus: 61,597 annotated pairs, roughly 12% of SNLI.
    • Training set: 55,517 pairs, including 39,655 neutral pairs (5 per premise from 7,931 premises).
    • Dev and test sets: 3,040 pairs each.
    • Annotation used 2–3 annotators per pair and a 10,000-step slider with a logistic rescaling.
  • BERT regressor trained only on u-SNLI: Pearson r = 0.6271 on test. A hypothesis-only baseline reaches about 0.41, which suggests SNLI's annotation artifacts carry over to u-SNLI.
  • Pre-training on about 500K mapped SNLI examples:
    • The label mapping was ENT→0.927, NEU→0.425, CON→0.021.
    • SNLI alone performs poorly (test r ≈ 0.50).
    • Adding u-SNLI fine-tuning raises results to dev r = 0.6762 and test r = 0.6589 / MSE = 0.0733.
  • Human baseline on dev: r = 0.6978, ρ = 0.7273, MSE = 0.0759. The best model approaches this but fails badly on some commonsense cases.
  • Availability: the data is released at nlp.jhu.edu/unli.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

53 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.774

@inproceedings{chen2020uncertain,
  title = {{Uncertain Natural Language Inference}},
  author = {Chen, Tongfei and Jiang, Zhengping and Poliak, Adam and Sakaguchi, Keisuke and Durme, Benjamin Van},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.774},
  url = {https://doi.org/10.18653/v1/2020.acl-main.774}
}

References

38 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Philippe Bernardy, Rasmus Blanck, Stergios Chatzikyriakidis, and Shalom Lappin. 2018. A compositional Bayesian semantics for natural language. In Proceedings of the First International Workshop on Language Cognition and Computational Models, pages 1–10. Association for Computational Linguistics. find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  3. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report, The FraCaS Consortium. find
  4. Robin Cooper, Simon Dobnik, Shalom Lappin, and Stefan Larsson. 2015. Probabilistic type theory and natural language semantics. Linguistic Issues in Language Technology, 10(1):1–43. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4711–4716. PASCAL Machine Learning Challenges Workshop, pages 177–190. find
  6. Donald Davidson. 1967. Truth and meaning. Synthese, 17(1):304–323. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4171–4186. find
  8. Jan van Eijck and Shalom Lappin. 2014. Probabilistic semantics for natural language. In Zoe Christoff, Paulo Galeazzi, Nina Gierasimczuk, Alexandru Marcoci, and Sonja Smets, editors, The Logic and Interactive Rationality Yearbook 2012, volume II. find
  9. Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. A probabilistic classification approach for lexical textual entailment. In Proc. AAAI, AAAI’05, pages 1050–1055. AAAI Press. find
  10. Noah D. Goodman and Daniel Lassiter. 2015. Probabilistic semantics and pragmatics: Uncertainty in language and thought. In Shalom Lappin and Chris Fox, editors, The Handbook of Contemporary Semantic Theory, 2nd edition. find
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2, pages 107– 112. find
  12. Daniel Kahneman and Amos Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–292. find
  13. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI. find
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. find
  15. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, Volume 1, pages 100–109. find
  16. Alice Lai and Julia Hockenmaier. 2017. Learning to predict denotational probabilities for modeling entailment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Volume 1, pages 721–730. find
  17. John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. 2018. Understanding deep learning performance through an examination of test set difficulty: John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648–657. find
  18. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2015, pages 1643–1648. find
  19. Zhongyang Li, Tongfei Chen, and Benjamin Van Durme. 2019. Learning to rank for plausible plausibility. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4818–4823. find
  20. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. The SICK (Sentences Involving Compositional Knowledge) dataset for relatedness and entailment. find
  21. Richard Montague. 1973. The proper treatment of quantification in ordinary english. In K. J. J. Hintikka, J. M. E. Moravcsik, and P. Suppes, editors, Approaches to Natural Language: Proceedings of the 1970 Stanford Workshop on Grammar and Semantics, pages 221–242. Springer Netherlands, Dordrecht. find
  22. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849. find
  23. Ellie Pavlick and Chris Callison-Burch. 2016. Most "babies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1, pages 2164–2173. find
  24. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Trans. Assoc. Comput. Linguistics, 7:677–694. find our card
  25. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088. ↗ our card
  26. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. find
  27. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Symposium. find
  28. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Trans. Assoc. Comput. Linguistics, 5:379– 395. find our card
  29. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 731–744. find our card
  30. Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1, pages 208–218. find
  31. Amram Shapiro, Louise Firth Campbell, and Rosalind Wright. 2014. Book of Odds: From Lightning Strikes to Love at First Sight, the Odds of Everyday Life. William Morrow Paperbacks. find
  32. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 2, pages 352–357. find
  33. Adam R. Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R. Gormley. 2017. Semantic proto-role labeling. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 4459–4466. find
  34. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation. find
  35. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science, 211(4481):453–458. ↗
  36. Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297– 323. find
  37. Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. Computational Linguistics, 43(4). find
  38. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1112–1122. find