Elamur Research⌕
☰
home / papers / 2018 / Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

Paper · 2018 · Atticus Geiger; Ignacio Cases; Lauri Karttunen; and Christopher Potts

Summary

Benchmarks built from naturalistic NLI corpora say little about how much compositional meaning a model actually captures, because their examples have no reliable measure of semantic difficulty. The authors build a controllable alternative. An interpreted grammar produces sentences with two quantifiers, optional modifiers and optional negation. Its open-class words are deliberately made logically independent of one another, so all of the difficulty comes from function words and composition. Each sentence pair is translated into first-order logic and labeled entailment, contradiction or neutral by the Prover9 theorem prover. The pairs are then resampled to balance both the lexical relations and the labels.

Five architectures are trained on 500K examples: CBoW, an LSTM encoder, a TreeNN, an Attention LSTM, and CompTreeNN, which reads premise and hypothesis as one aligned tree. All models except CompTreeNN stall near 96% accuracy. They learn quantifiers, negation and which slots are empty, but they lose track of which content words appear. CompTreeNN is nearly perfect. The study shows how synthetic, fully interpretable data can reveal architectural information bottlenecks that aggregate scores hide.

Key points

  • The fragment follows the template Q Adj N (Neg) (Adv) V Q Adj Adj N.

    • Quantifiers are every, not every, some and no.
    • Each open-class category has 100 words.
    • Empty modifier slots are marked by a special token.
    • Scope follows surface order.

  • Labels come from Prover9, with added premises stating that no expression is empty or universal. Relations between aligned phrases follow MacCartney and Manning's seven basic relations.

  • Without rebalancing, about 98% of generated pairs would be neutral. The data are resampled so that adjective–noun and adverb–verb pairs are equally likely to be equal, subset, superset or independent. Generation code is published as MultiplyQuantifiedData on GitHub.

  • Test accuracy (mean of 5 runs) and accuracy on the "informative open-class" subset:

    ModelTest accuracyInformative open-class subset
    CBoW69.6682.30
    LSTM95.6126.90
    TreeNN95.9931.09
    Attention LSTM95.8235.69
    CompTreeNN99.8598.05

    The informative open-class subset contains neutral items that would flip to entailment or contradiction if the aligned content words were identical.

  • The authors argue the failures are architectural rather than a tuning problem:

    • Doubling the embedding size to 200 changes nothing.
    • A broad hyperparameter sweep for the LSTM produces only two clusters of runs: fully stuck, or partly recovered.

Interesting ideas

  • I1. Make content words mutually independent on purpose, so that any error must come from composition and function words, not from lexical knowledge. This yields a clean probe of one skill.
  • I2. Produce gold labels by mapping sentences deterministically into logic and running an off-the-shelf prover, which gives unlimited, perfectly labeled pairs with known complexity.
  • I3. In this fragment, the parity of negations and negative quantifiers alone separates contradiction from entailment, a cheap invariant that any correct system must respect.
  • I4. Align premise and hypothesis node by node, decide a local relation at each leaf, and propagate the relations up the tree, as in natural logic; this avoids squeezing each sentence into a single vector.
  • I5. Report accuracy on a targeted subset that isolates one known failure mode; a near-ceiling overall score hid a collapse to roughly 30% there.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build diagnostic suites where the nouns and verbs are nonce or unrelated, so that quantifier, negation and scope handling in the pipeline is tested on its own.
  • I2 (world): Compile the world of a text into first-order facts and use a prover as an oracle that checks whether Mova's answers actually follow.
  • I3 (answer): Add parity-style consistency checks that flag answers whose polarity disagrees with the count of negative operators on the reasoning path.
  • I4 (frames): Compare a question's UD tree with the matching text tree through aligned subconstituents, using the seven relations, so that every inference step stays explainable.
  • I5 (learn): Pair every global metric with negative-control subsets built to catch specific shortcuts, such as ignoring which entity is named.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Atticus Geiger; Ignacio Cases; Lauri Karttunen; and Christopher Potts. 2018. Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences. arXiv:1810.13033.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.13033 · arXiv 1810.13033

@misc{karttunen2018stress,
  title = {{Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences}},
  author = {Atticus Geiger; Ignacio Cases; Lauri Karttunen; and Christopher Potts},
  year = {2018},
  doi = {10.48550/arxiv.1810.13033},
  eprint = {1810.13033},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.13033}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  2. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015b. Recursive neural networks can learn logical semantics. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 12–21. Association for Computational Linguistics. find
  3. George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314. ↗
  4. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. ↗
  5. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. Association for Computational Linguistics.
  6. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics. find
  7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  8. Thomas F. Icard and Lawrence S. Moss. 2013. Recent progress on monotonicity. Linguistic Issues in Language Technology, 9(7):1–31. ↗
  9. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Bill MacCartney. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. ↗
  10. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140–156. Association for Computational Linguistics. find
  11. W. McCune. 2005–2010. Prover9 and Mace4. http://www.cs.unm.edu/˜mccune/prover9/. ↗
  12. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  13. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. Association for Computational Linguistics. find
  14. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. CoRR, abs/1509.06664. ↗
  15. Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 74–79. Association for Computational Linguistics.
  16. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011a. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 151–161, Stroudsburg, PA, USA. Association for Computational Linguistics.
  17. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 151–161, Stroudsburg, PA, USA. Association for Computational Linguistics.
  18. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. CoRR, abs/1804.08117. ↗
  19. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geiger-2018-stress-testing-neural-models-natural/