Elamur Research⌕
☰
home / papers / 2021 / Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference

Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference

Paper · 2021 · Qingyuan Hu, Yi Zhang, Kanishka Misra et al.

Summary

NLI benchmarks are meant to test whether a system can relate a premise to a hypothesis. Earlier work by Poliak et al. showed that models which see only the hypothesis can still perform surprisingly well. That suggests datasets carry statistical giveaways unrelated to inference. The papers that build automatically recast datasets have not examined these giveaways closely.

This paper looks at one such recast resource, the Semantic Proto-Roles (SPR) dataset from White et al. It trains hypothesis-only models on SPR and inspects which individual words push predictions towards a label. On this data, hypothesis-only baselines had already been reported to clearly beat the majority baseline and to approach the then state of the art, a modified InferSent. The authors find signs of lexical biases in the hypotheses that could inflate reported accuracy, so models may be "right for the wrong reasons".

The aim is diagnosis, not a stronger model. The work warns that template-generated inference data needs word-level auditing before its scores are trusted as evidence of understanding.

Key points

  • Poliak et al. compared hypothesis-only baselines across 10 NLI datasets. They grouped the datasets into three kinds:
    • human-elicited, such as SNLI;
    • human-judged, such as SciTail;
    • automatically recast, such as SPR, DPR and FN+.
  • Earlier findings cited by the paper:
    • in SNLI, negation words such as "nobody" strongly signal contradiction;
    • in recast FN+, more grammatical hypotheses tend to be labelled entailed.
  • SPR breaks semantic roles into 16 finer properties, such as instigation. Annotators rated how likely each property was on a 1–5 scale. Scores of 1–3 became "not entailed" and 4–5 became "entailed", with no neutral class.
  • The reference model is InferSent, a BiLSTM sentence encoder with pooling. Hypothesis-only models on SPR came close to its performance.
  • Future work proposed by the authors: test whether existing models actually exploit the found biases by comparing their outputs on biased versus debiased instances.

Interesting ideas

  • I1. Partial-input baselines: feed a model only part of the input (here, the hypothesis without the premise). If accuracy stays high, the benchmark leaks the answer through surface cues.
  • I2. Word-level label scan: check, word by word, how strongly each word's presence predicts a label. This cheaply surfaces giveaway tokens in automatically produced data.
  • I3. Recasting templates leave fingerprints: converting a semantic resource into inference pairs by template produces repeated phrasing, which models can latch onto instead of meaning.
  • I4. Roles as bundles of graded properties: SPR describes an argument through likelihoods of properties like instigation rather than a single role label. That gives finer, more explainable role judgements.
  • I5. Surface quality as a confound: in FN+, how grammatical a hypothesis was correlated with its label. Text quality alone can therefore act as a hidden signal.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Add "question-only" and "text-shuffled" runs as standing negative controls in evaluation. A reader that still scores well without the story reveals a leaky test set.
  • I2 (learn): Run this scan over silver data and LLM-teacher-generated items before training. Flag words whose label distribution is extreme so they can be balanced or removed.
  • I3 (answer): When generating school-course tests or fable QA from templates, vary the wording and check for template-specific cues. This keeps reader scores reflecting real comprehension.
  • I4 (frames): Alongside VerbNet roles, frames could store proto-role properties (caused it, changed state, etc.). This would make who-did-what-to-whom decisions more transparent.
  • I5 (back): When generated sentences are used as test items or training pairs, keep fluency comparable across true and false variants. That prevents quality from betraying the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Purdue University West LafayetteWest Lafayette · United States9 works · 1,254 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Qingyuan Hu, Yi Zhang, Kanishka Misra and Julia Rayz. 2021. Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference. arXiv:2101.07397.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2101.07397 · arXiv 2101.07397

@misc{hu2021exploring,
  title = {{Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference}},
  author = {Qingyuan Hu, Yi Zhang, Kanishka Misra and Julia Rayz},
  year = {2021},
  doi = {10.48550/arxiv.2101.07397},
  eprint = {2101.07397},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2101.07397}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” arXiv:1508.05326 [cs], Aug. 2015, arXiv: 1508.05326. [Online]. Available: http://arxiv.org/abs/1508.05326 ↗
  2. A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme, “Hypothesis Only Baselines in Natural Language Inference,” in Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. New Orleans, Louisiana: Association for Computational Linguistics, 2018, pp. 180–191. [Online]. Available: http://aclweb.org/anthology/S18-2023 ↗
  3. I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, ser. Lecture Notes in Computer Science, J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d’Alché Buc, Eds. Berlin, Heidelberg: Springer, 2006, pp. 177–190. find
  4. T. Khot, A. Sabharwal, and P. Clark, “Scitail: A textual entailment dataset from science question answering,” 2018. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17368 ↗
  5. A. S. White, P. Rastogi, K. Duh, and B. Van Durme, “Inference is everything: Recasting semantic resources into a unified evaluation framework,” in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 996–1005. our card
  6. T. McCoy, E. Pavlick, and T. Linzen, “Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 3428–3448. our card
  7. J. J. Katz, Semantic Theory. New York: Harper & Row, 1972. find
  8. S. Bowman and X. Zhu, “Deep learning for natural language inference,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 6–8. [Online]. Available: https://www.aclweb.org/anthology/N19-5002 ↗
  9. S. Wang and J. Jiang, “Learning natural language inference with lstm,” arXiv preprint arXiv:1512.08849, 2015. ↗ our card
  10. O. Glickman and I. Dagan, “Web based probabilistic textual entailment,” in In Proceedings of the 1st Pascal Challenge Workshop, 2005. find
  11. Y. Mehdad, A. Moschitti, and F. M. Zanzotto, “Semker: Syntactic/semantic kernels for recognizing textual entailment,” Theory and Applications of Categories, 2009. find
  12. B. MacCartney, M. Galley, and C. D. Manning, “A phrase-based alignment model for natural language inference,” in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 2008, pp. 802–811. find
  13. P. Clark and P. Harrison, “An inference-based approach to recognizing entailment,” Theory and Applications of Categories, 2009. find
  14. H. Wang, D. Sun, and E. P. Xing, “What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 7136–7143. find
  15. H. Wang, A. Meghawat, L. Morency, and E. P. Xing, “Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis,” CoRR, vol. abs/1609.05244, 2016. [Online]. Available: http://arxiv.org/abs/1609.05244 ↗
  16. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” CoRR, vol. abs/1612.00837, 2016. [Online]. Available: http://arxiv.org/abs/1612.00837 ↗ our card
  17. T. Yue and H. Wang, “Deep learning for genomics: A concise overview,” 2018. ↗
  18. R. Cooper, D. Crouch, J. V. Eijck, C. Fox, J. V. Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, and K. Konrad, “Using the framework,” 1996. find
  19. D. Reisinger, R. Rudinger, F. Ferraro, C. Harman, K. Rawlins, and B. Van Durme, “Semantic proto-roles,” Transactions of the Association for Computational Linguistics, vol. 3, pp. 475–488, 2015. ↗ our card
  20. E. Pavlick, T. Wolfe, P. Rastogi, C. Callison-Burch, M. Dredze, and B. Van Durme, “FrameNet+: Fast paraphrastic tripling of FrameNet,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Beijing, China: Association for Computational Linguistics, Jul. 2015, pp. 408– 413. [Online]. Available: https://www.aclweb.org/anthology/P15-2067 ↗ our card
  21. A. Rahman and V. Ng, “Resolving complex cases of definite pronouns: The Winograd schema challenge,” in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Jeju Island, Korea: Association for Computational Linguistics, Jul. 2012, pp. 777–789. [Online]. Available: https://www.aclweb.org/anthology/D12-1071 ↗ our card
  22. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” CoRR, vol. abs/1705.02364, 2017. [Online]. Available: http://arxiv.org/abs/1705.02364 ↗ our card
  23. A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2018, pp. 1112–1122. [Online]. Available: http://aclweb.org/anthology/N18-1101 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2021-exploring-lexical-irregularities-hypothesis-only/