Elamur Research⌕
☰
home / papers / 2021 / FANCY: A Diagnostic Data-Set for NLI Models

FANCY: A Diagnostic Data-Set for NLI Models

Paper · 2021 · Guido Rocchietti, Flavia Achena, Giuseppe Marziano et al.

Summary

FANCY addresses the gap between aggregate benchmark scores and actual linguistic competence in Natural Language Inference (NLI) models. The authors hand-built 4,000 short, deliberately simple premise–hypothesis pairs covering four phenomena: factivity, negation, hierarchical lexical relations (hypernymy/hyponymy under monotonicity), and common-sense knowledge. Each pair has a coarse NLI label (entailment, neutral, contradiction) and fine-grained sub-category tags, following the layout of the SuperGLUE diagnostic set. They evaluated bert-base-uncased-MNLI, roberta-large-mnli and a GloVe-based CBOW baseline, all trained on MNLI. Every model scored well below its MNLI level. RoBERTa was strongest and did comparatively well on negation and factivity. It struggled with hierarchical relations, especially downward-monotone contexts, and with common-sense items involving time, preconditions and space. Many of its failures were on sentences that people find trivial. The dataset therefore works as a diagnostic tool: it shows which specific inference types a model lacks, which a single benchmark number cannot do.

Key points

  • The dataset has 4,000 manually written pairs: 1,406 entailment, 1,622 neutral and 968 contradiction. A column records which sentence serves as premise, so a pair can be read in either direction.
  • Accuracy by subset (MNLI / factivity / negation / hierarchical / common-sense):
    • RoBERTa: 90.2 / 74.6 / 82.0 / 60.4 / 68.0
    • BERT: 84.6 / 65.2 / 70.0 / 49.7 / 57.0
    • CBOW: 65.2 / 45.1 / 45.0 / 37.8 / 41.0
  • Factivity errors: RoBERTa's mistakes cluster on "Possibly Fact" hypotheses that should be neutral, with 257 errors out of 416 items.
  • Hierarchical errors: downward-monotone items are by far the worst, with 189 errors out of 222. Non-monotone items are also hard.
  • Negation errors: the hardest cases involve quantifiers, modals, temporal adverbs and relative pronouns. Common-sense errors concentrate on temporal relations, preconditions and spatial relations.

Interesting ideas

  • I1. Each item carries both a three-way NLI label and a fine sub-phenomenon tag, so a single accuracy score can be broken down into a map of specific weaknesses.
  • I2. Hypernymy entailment is tested together with the monotonicity of its context: "some parrots" licenses "some birds", but "no fish" licenses "no carp", and "most" licenses neither. This separates knowing a lexical relation from projecting it correctly through a sentence.
  • I3. Factive verbs ("know") are contrasted with non-factive ones ("think, believe"), and the status of the embedded event is labelled as fact, possible fact, counterfact and so on. The model's main failure was treating mere possibilities as facts.
  • I4. Negation is split into implicit ("forget"), affixal ("illegal"), contrastive ("not coffee but tea"), partial-scope and quantifier-interacting types. This turns a vague phenomenon into a checklist of separately testable cases.
  • I5. Items are kept deliberately simple so that failures can be attributed to the phenomenon itself rather than to syntactic or lexical noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag our evaluation items with the phenomenon they test, so a regression report says which inference type broke rather than only that the score dropped.
  • I2 (world): Store a monotonicity signature for each quantifier and negation in the compiled knowledge, so hypernym inferences from WordNet are flipped or blocked according to the context they appear in.
  • I3 (frames): Give each embedded clause a factuality value derived from the governing verb, so the world of the text does not record a believed proposition as true.
  • I4 (tree): Build small regression suites per negation type to verify that the parse attaches the negator to its correct scope, for example to an adverb rather than to the verb.
  • I5 (answer): Write minimal, clean probe questions alongside the fable QA tests, so that an error points to one missing reasoning ability instead of to parsing difficulty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Guido Rocchietti, Flavia Achena, Giuseppe Marziano, Sara Salaris and Alessandro Lenci. 2021. FANCY: A Diagnostic Data-Set for NLI Models. Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rocchietti2021fancy,
  title = {{FANCY: A Diagnostic Data-Set for NLI Models}},
  author = {Guido Rocchietti, Flavia Achena, Giuseppe Marziano, Sara Salaris and Alessandro Lenci},
  booktitle = {Proceedings of the Eighth Italian Conference on Computational Linguistics (CLiC-it 2021)},
  year = {2021},
  doi = {10.4000/books.aaccademia.10804},
  url = {https://doi.org/10.4000/books.aaccademia.10804}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Trans. Assoc. Comput. Linguist., 7:49–72. ↗
  2. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Durme, and Alexander Rush. 2019. On adversarial removal of hypothesis-only bias in natural language inference. pages 256–262.
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. ↗
  4. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136. Association for Computational Linguistics. ↗
  5. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
  6. Richard Feldman. 2003. Epistemology. Prentice Hall. find
  7. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics.
  8. Laurence Horn. 1989. A Natural History of Negation. University of Chicago Press. find
  9. Ray S. Jackendoff. 1969. An interpretive theory of negation. Foundations of Language, 5(2):218–241. Publisher: Springer. find
  10. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. ↗
  11. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. ↗
  12. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019a. Inoculation by fine-tuning: A method for analyzing challenge datasets. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692 [cs]. ↗
  14. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140–156, Tilburg, The Netherlands. Association for Computational Linguistics. ↗
  15. Bill MacCartney and Christopher D. Manning. 2014. Natural logic and natural language inference. In Computing Meaning, volume 47, pages 129–147. Springer Netherlands. ↗
  16. James D. McCawley. 1991. The Syntactic Phenomena of English. find
  17. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. Association for Computational Linguistics. ↗
  18. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  19. James Pustejovsky and Olga Batiukova. 2019. The Lexicon. Cambridge University Press, Cambridge, England. find
  20. Stephen Creig Roller. 2017. Identifying lexical relationships and entailments with distributional semantics. ↗
  21. Roser Saurı́ and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational Linguistics -COLI, 38:1–39.
  22. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. ↗
  23. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in neural information processing systems, volume 32. Curran Associates, Inc.
  24. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rocchietti-2021-fancy-diagnostic-data-set-nli/