Elamur Research⌕
☰
home / papers / 2018 / Analyzing Compositionality-Sensitivity of NLI Models

Analyzing Compositionality-Sensitivity of NLI Models

Paper · 2018 · Yixin Nie, Yicheng Wang and Mohit Bansal

Summary

NLI systems reach near-human accuracy on SNLI and MNLI. This paper asks whether they use sentence structure or mostly match words. The authors test seven strong models, both sentence encoders and co-attention models. They build two rule-based adversaries that keep the vocabulary of the premise but change its meaning. On these, every model fails badly. Two more experiments support this: models whose RNNs are replaced by fully-connected layers, or that are trained on shuffled words, still score well on the standard test sets. Adversarial training is also too narrow, because learning to handle one adversary type does not carry over to another.

As a broader tool, the authors propose compositionality-sensitivity (CS) testing. A bag-of-words regression model scores each natural test example with a lexically-misleading score (LMS), and evaluation keeps only examples where that model confidently prefers a wrong label. On these subsets, models that look equal on standard benchmarks separate by architecture: bag-of-words models do worst, sequential models do better, and syntax-tree models do best. The authors release the code and data.

Key points

  • Seven models were evaluated: RSE, G-TLSTM, DAM, ESIM, S-TLSTM, DIIN and DR-BiLSTM. Their SNLI accuracies range from about 85% to 88.3%.
  • SOSWAP adversary: it swaps subject and object in SNLI premises with subject-verb-object structure, producing contradictions. It yielded 971 examples, built using dependency parses from the Stanford Parser (CoreNLP 3.8.0).
  • ADDAMOD adversary: it adds the same adjective to different nouns in the premise and hypothesis, producing neutral pairs.
  • On these adversaries, models overwhelmingly predict entailment. DAM, for example, labels about 99.7% of SOSWAP pairs as entailment.
  • Removing composition costs little on standard data. ESIM falls only from 88.17 to 86.79 on SNLI with shuffled training words, and RSE (BoW) still reaches 85.02. On the CS subsets, models drop sharply while humans stay at roughly 80–89%.

Interesting ideas

  • I1. Build a hard test set by keeping only examples where a bag-of-words model is confidently wrong; this filters natural data, so no synthetic sentences are needed, and it isolates cases that require structure.
  • I2. Adversaries that keep the words unchanged but alter meaning, such as swapping subject and object or moving a modifier to another noun, expose models that ignore who did what to whom.
  • I3. Ablate structure deliberately, by shuffling input words or deleting the sequence model, and check whether the score drops; if it does not, the benchmark cannot measure structural understanding.
  • I4. A model can solve one targeted phenomenon, such as double negation, through a lexical shortcut, so success on a diagnostic item should be checked against what a bag-of-words model could do.
  • I5. Training on one adversary type causes overfitting rather than general robustness, so evaluation should span several unrelated perturbation families.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use a bag-of-words baseline as a negative control, scoring Mova's accuracy separately on items where that baseline is confidently wrong, to show the tree-based pipeline earns its keep.
  • I2 (frames): Generate role-swapped and modifier-moved variants of parsed sentences and check that the semantic-role frames flip correctly, giving cheap regression tests for the role assigner.
  • I3 (answer): Feed word-shuffled school tests and fable questions to the readers; if accuracy barely changes, the test items are lexically solvable and need replacing.
  • I4 (learn): Attach a lexical-shortcut score to each diagnostic or self-play item so that passing easy items is not taken as proof that a rule was learned.
  • I5 (learn): When inducing repair rules from one error type, hold out a different perturbation family to confirm the rules generalize rather than memorize the pattern.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States37 works · 14,332 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Yixin Nie, Yicheng Wang and Mohit Bansal. 2018. Analyzing Compositionality-Sensitivity of NLI Models. arXiv:1811.07033.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.07033 · arXiv 1811.07033

@misc{nie2018analyzing,
  title = {{Analyzing Compositionality-Sensitivity of NLI Models}},
  author = {Yixin Nie, Yicheng Wang and Mohit Bansal},
  year = {2018},
  doi = {10.48550/arxiv.1811.07033},
  eprint = {1811.07033},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.07033}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP. ↗
  2. Carmona, V. I. S.; Mitchell, J.; and Riedel, S. 2018. Behavior analysis of nli models: Uncovering the influence of three factors on robustness. NAACL. ↗ our card
  3. Chen, D., and Manning, C. 2014. A fast and accurate dependency parser using neural networks. In EMNLP. our card
  4. Chen, Q.; Zhu, X.; Ling, Z.-H.; Wei, S.; Jiang, H.; and Inkpen, D. 2017. Enhanced lstm for natural language inference. In ACL. find
  5. Choi, J.; Yoo, K. M.; and Lee, S.-g. 2017. Learning to compose task-specific tree structures. In AAAI. ↗
  6. Dasgupta, I.; Guo, D.; Stuhlmüller, A.; Gershman, S. J.; and Goodman, N. D. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302. ↗
  7. Evans, R.; Saxton, D.; Amos, D.; Kohli, P.; and Grefenstette, E. 2018. Can neural networks understand logical entailment? ICLR. find
  8. Ghaeini, R.; Hasan, S. A.; Datla, V.; Liu, J.; Lee, K.; Qadir, A.; Ling, Y.; Prakash, A.; Fern, X. Z.; and Farri, O. 2018. Dr-bilstm: Dependent reading bidirectional lstm for natural language inference. NAACL. find
  9. Glockner, M.; Shwartz, V.; and Goldberg, Y. 2018. Breaking nli systems with sentences that require simple lexical inferences. NAACL. our card
  10. Gong, Y.; Luo, H.; and Zhang, J. 2017. Natural language inference over interaction space. ICLR. ↗
  11. Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S. R.; and Smith, N. A. 2018. Annotation artifacts in natural language inference data. NAACL. find
  12. Jia, R., and Liang, P. 2017. Adversarial examples for evaluating reading comprehension systems. EMNLP. our card
  13. Nangia, N., and Bowman, S. R. 2018. Listops: A diagnostic dataset for latent tree learning. ACL-SRW. find
  14. Nangia, N.; Williams, A.; Lazaridou, A.; and Bowman, S. R. 2017. The repeval 2017 shared task: Multi-genre natural language inference with sentence representations. RepEval. our card
  15. Nie, Y., and Bansal, M. 2017. Shortcut-stacked sentence encoders for multi-domain inference. RepEval. find
  16. Parikh, A. P.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016. A decomposable attention model for natural language inference. In EMNLP. ↗ our card
  17. Poliak, A.; Haldar, A.; Rudinger, R.; Hu, J. E.; Pavlick, E.; White, A. S.; and Van Durme, B. 2018a. Towards a unified natural language inference framework to evaluate sentence representations. arXiv preprint arXiv:1804.08207. ↗ our card
  18. Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Van Durme, B. 2018b. Hypothesis only baselines in natural language inference. *SEM. find
  19. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. Squad: 100,000+ questions for machine comprehension of text. EMNLP. ↗ our card
  20. Tsuchiya, M. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. LREC. our card
  21. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP. ↗ our card
  22. Wang, Z.; Hamza, W.; and Florian, R. 2017. Bilateral multi-perspective matching for natural language sentences. IJCAI. our card
  23. Williams, A.; Nangia, N.; and Bowman, S. R. 2018. A broad-coverage challenge corpus for sentence understanding through inference. NAACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/