Elamur Research⌕
☰
home / papers / 2021 / Exploring Transitivity in Neural NLI Models through Veridicality

Exploring Transitivity in Neural NLI Models through Veridicality

Paper · 2021 · Hitomi Yanaka, Koji Mineshima and Kentaro Inui

Summary

The paper asks whether neural NLI models can chain inferences: if a model has learned that A entails B and that B entails C, can it conclude that A entails C without having seen that combination? The authors probe this through veridicality. Factive verbs like know pass the truth of their complement clause up to the sentence, while verbs like hope or believe do not. Embedding a clause inside such a verb lets one build composite inference problems at scale. Two datasets were built and released. A synthetic one pairs veridical verbs with Boolean inferences, and a naturalistic one pairs them with lexical and structural inferences. Models are trained on the two basic patterns and tested on their compositions. LSTM and BERT models consistently fail on the combinations. Adding some composite examples to training helps, but only because the test items resemble the added items, not because the models compose. Crowdsourced humans follow transitivity far better, though they have their own blind spot. The work shows that high benchmark accuracy can hide a missing ability to compose inferences.

Key points

  • The setup has three patterns. Basic pattern I1 is f(s1) → s1, a verb with an embedded clause. Basic pattern I2 is s1 → s2, an inference from the clause. The composite f(s1) → s2 gets its gold label from the transitivity of entailment. Labels are binary: yes (entailment) and unk (non-entailment).
  • On composites with a veridical verb and a non-entailing embedded step, both models do very badly when trained only on basic patterns:
    • LSTM scores 0.0% and BERT 8.9%, against 98.8% for humans.
    • Overall accuracy is 72.9% for LSTM, 77.2% for BERT and 85.5% for humans.
  • Augmentation with composite examples was tested by adding veridical verbs one at a time. BERT reached 100% on the test set after adding composites from four verbs, while LSTM needed twelve. Accuracy also rose when the basic I2 examples were removed from training, which points to similarity matching rather than composition.
  • Human judgements were collected for 960 naturalistic items on Amazon Mechanical Turk:
    • Each item received three ratings on a 5-point Likert scale, with inter-rater Pearson correlation of 0.76.
    • Ratings were binarised (4 or higher = yes) and decided by majority vote.
    • Humans scored only 44.9% when the verb is non-veridical but the embedded step is entailing. The authors attribute this to veridicality bias.
  • Data and code are public at github.com/verypluming/transitivity.

Interesting ideas

  • I1. Build composite test items by wrapping one inference inside another (a clause-embedding verb around an already-labelled pair), so the gold label follows from the parts automatically and difficulty scales without new annotation.
  • I2. Store a per-verb veridicality flag and propagate truth through the embedding chain, instead of letting surface overlap decide. The paper's know that Ann or Bob left case shows that lexical overlap misleads here.
  • I3. Check whether improvement after augmentation is real composition by removing one basic ingredient from training. If scores still rise, the model is matching similar items, not composing.
  • I4. Collect graded human judgements and binarise them with an explicit threshold, so human and system errors can be compared per cell of the label table. This exposed a human bias toward treating complements as true even after believe.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): The school-test and fable QA readers could get auto-generated chained questions (e.g. "X knew that Y gave Z…") whose expected answers are derived from simpler, already-verified items.
  • I2 (world): Mark complements of non-veridical verbs as unasserted or belief-only states in the text world, so the answer step cannot treat them as facts just because the words match.
  • I3 (learn): Use this ablation as a negative control when self-induced repair rules or new silver data raise scores, to confirm the gain comes from the intended mechanism.
  • I4 (frames): When the pragmatics layer models implicature, it could optionally predict the human-like veridicality bias alongside the strict reading, separating literal entailment from likely reader inference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tohoku UniversitySendai · Japan9 works · 363 citations🎓Keio UniversityTokyo · Japan4 works · 705 citations🔬RIKENWako · Japan8 works · 412 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Hitomi Yanaka, Koji Mineshima and Kentaro Inui. 2021. Exploring Transitivity in Neural NLI Models through Veridicality. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.78

@inproceedings{yanaka2021exploring,
  title = {{Exploring Transitivity in Neural NLI Models through Veridicality}},
  author = {Hitomi Yanaka, Koji Mineshima and Kentaro Inui},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.78},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.78}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 632–642. ↗
  2. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015b. Recursive neural networks can learn logical semantics. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 12–21. find
  3. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Proceedings of the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence (IJCAI-PRICAI).
  4. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. ↗
  5. Dirk van Dalen. 2013. Logic and Structure, 5 edition. Springer. find
  6. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 1596– 1601. ↗
  7. Jacob Devlin, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186.
  8. Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71. ↗
  9. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2019. Posing fair generalization tasks for natural language inference. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4484–4494. ↗
  10. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173.
  11. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 650–655.
  12. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1958–1969. find
  13. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 107– 112. find
  14. Irene Heim and Angelika Kratzer. 1998. Semantics in Generative Grammar. Blackwell. find
  15. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). ↗
  16. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1426–1436.
  17. Brenden M. Lake and Marco Baroni. 2017. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the International Conference on Machine Learning (ICML). ↗
  18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics (TACL), 4:521–535. ↗
  19. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2171– 2179.
  20. Gary Marcus. 2003. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press. find
  21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC), pages 216–223.
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. ↗
  23. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301–333.
  24. Philip N. Johnson-Laird and Ruth M.J. Byrne. 1991. Deduction. Erlbaum. find
  25. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3428– 3448.
  26. Lauri Karttunen and Stanley Peters. 1979. Conventional implicatures. In Choon Kyu Oh and David A. Dineen, editors, Syntax and Semantics 11: Presupposition, pages 1–56. Academic Press. find
  27. Sangeet Khemlani and Philip N Johnson-Laird. 2012. Theories of the syllogism: A meta-analysis. Psychological bulletin, 138(3):427–457. ↗
  28. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105. find
  29. Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2339–2352.
  30. Koji Mineshima, Pascual Martínez-Gómez, Yusuke Miyao, and Daisuke Bekki. 2015. Higher-order logical inference with compositional semantics. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2055–2061. find
  31. Richard Montague. 1973. The proper treatment of quantification in ordinary English. In Jaakko Hintikka, Julius M. E. Moravcsik, and Patrick Suppes, editors, Approaches to Natural Language, pages 189–224. Reidel, Dordrecht. Reprinted in Richmond H. Thomason (ed.), Formal Philosophy: Selected Papers of Richard Montague, 247–270, 1974, New Haven: Yale University Press. ↗
  32. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. Analyzing compositionality-sensitivity of NLI models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6867–6874.
  33. Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1027–1037. find
  34. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. ↗
  35. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM), pages 180–191. find
  36. Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence. ↗
  37. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240. ↗
  38. Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196–205. find
  39. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4506–4515. find
  40. Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4911–4921.
  41. Anne S. Troelstra and Helmut Schwichtenberg. 2000. Basic Proof Theory, 2 edition. Cambridge University Press. find
  42. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC).
  43. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of Advances in Neural Information Processing Systems 32 (NIPS), pages 3266–3280.
  44. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR). ↗
  45. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 740–745.
  46. Aaron Steven White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, Amherst, MA, USA. GLSA Publications. find
  47. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4717–4724. ↗
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1112–1122. find
  49. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6105–6117. ↗
  50. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40. ↗
  51. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 250–255. ↗
  52. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics (TACL), 5:379–395. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yanaka-2021-exploring-transitivity-neural-nli-models/