Elamur Research⌕
☰
home / papers / 2023 / Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference

Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference

Paper · 2023 · Sushma Anand Akoju, Robert Vacareanu, Eduardo Blanco et al.

Summary

The paper examines whether transformer NLI models handle the compositional reasoning described by Natural Logic (MacCartney, 2009), especially how quantifiers, negation and modifiers shift monotonicity and therefore the entailment label. The authors build SICCK (Sentences Involving Complex Compositional Knowledge), a synthetic set of 1,304 premise/hypothesis pairs. It starts from 15 seed pairs from SICK. Parsers locate the subject, verb and object, and the authors attach universal quantifiers, existential quantifiers, negations, and adjectives or adverbs to those slots, in the premise, the hypothesis, or both. Each new pair is then labelled by hand using monotonicity calculus and MacCartney's seven relations. Neural NLI models perform poorly on this data, both zero-shot and after fine-tuning. Negation and existential quantifiers are the hardest cases, and verb modifications cause more errors than subject or object ones. Fine-tuning brings little gain. The work matters because it gives a small but tightly controlled probe showing that strong NLI scores do not imply logically compositional inference.

Key points

  • SICCK has 1,304 pairs built from 15 SICK seeds. The seeds split into 5 Entailment, 5 Neutral and 5 Contradiction pairs. The dataset is released at github.com/clulab/releases (branch sushma/acl2023-nlrse-sicck).
  • Modifier slots are found with parsers:
    • Subjects are the left-most NP from the Berkeley Neural Parser.
    • Verbs are the right-most VP from the same parser.
    • Objects are NPs attached to the main verb, found with the dependency parser of Vacareanu et al. (2020).
  • Labels use seven relations: FE, RE, Negation, Alternation, Cover, Independence and Equivalence. For evaluation these are compressed to four: FE, RE, Contradiction and Neutral. For example, changing the hypothesis subject "a man" to "every man" turns the seed pair's FE label into RE.
  • In the fine-tuning experiments, training lasted either 4 or 8 epochs, with a batch size of 8, 16 or 32. Accuracies fall roughly between 0.43 and 0.60.
  • The error analysis focuses on negation and verb edits:
    • Of 167 negation examples, 122 are Neutral, and models got only about 37% of those right.
    • Among the 220 verb-modified examples, 31 are Neutral with negation. On these, models swung between Contradiction and FE in 99% of cases.
    • About 69.4% of the 49 Neutral examples with universal modifiers on verbs were mispredicted.

Interesting ideas

  • I1. The dataset multiplies a handful of gold pairs by inserting logical operators at fixed syntactic slots and re-deriving each label from monotonicity rules, which gives a large, controlled test from tiny seeds.
  • I2. Insertion points come from parse structure (left-most NP, right-most VP, dependents of the main verb) rather than string positions, so the edits stay grammatical and target real constituents.
  • I3. Labelling uses the full seven-way Natural Logic inventory rather than 3-way NLI, so directional cases like RE, Cover and Alternation stay visible instead of collapsing into "neutral".
  • I4. Errors are broken down by modifier type, SVO slot and gold label, which exposed specific weak spots such as Neutral plus negation and verb edits that an overall score hides.
  • I5. Each modification is applied to the premise only, the hypothesis only, or both, which tests whether the model treats the two sides asymmetrically as logic requires.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): We could turn a few hand-checked fable sentences into hundreds of perturbed probes with rule-derived answers. These would serve as negative controls showing whether a component tracks the operator or just the words.
  • I2 (tree): The same slot-finding can stress-test the parser and gparse. After we insert "not", "every" or an adverb, the rest of the UD tree should stay stable, and any change is a detectable regression.
  • I3 (world): The world layer could store relations between propositions in this seven-way form. That would let quantifier and negation effects compose step by step, with each step explainable.
  • I4 (answer): The school-test and QA readers could report accuracy in the same grid. This would show whether failures cluster on negated verbs or existential subjects.
  • I5 (answer): QA on tales could include probes where the negation sits in the story versus in the question. This checks that the reader's polarity handling is directional.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ArizonaTucson · United States38 works · 10,603 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Sushma Anand Akoju, Robert Vacareanu, Eduardo Blanco, Haris Riaz and Mihai Surdeanu. 2023. Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference. Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.nlrse-1.12

@inproceedings{akoju2023synthetic,
  title = {{Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference}},
  author = {Sushma Anand Akoju, Robert Vacareanu, Eduardo Blanco, Haris Riaz and Mihai Surdeanu},
  booktitle = {Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)},
  year = {2023},
  doi = {10.18653/v1/2023.nlrse-1.12},
  url = {https://doi.org/10.18653/v1/2023.nlrse-1.12}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze and Johan Bos. 2017. Towards universal semantic tagging. arXiv preprint arXiv:1709.10381. ↗
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015b. The snli corpus. find
  4. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015c. Recursive neural networks can learn logical semantics. find
  5. Robin Cooper, R Crouch, Jan van Eijck, Chris Fox, Josef van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. the fracas consortium. Technical report, Technical report, FraCaS deliverable D-16. find
  6. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 107–114.
  9. Izumi Haruta, Koji Mineshima, and Daisuke Bekki. 2020. Logical inferences with comparatives and generalized quantifiers. find
  10. Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723, Dublin, Ireland. Association for Computational Linguistics. find
  11. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106–9118, Online. Association for Computational Linguistics. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  12. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗
  13. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. ↗
  14. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  15. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 79–86. Association for Computational Linguistics. ↗
  16. Ankur P. Parikh, Oscar T"ackstr"om, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. ArXiv, abs/1606.01933. ↗
  17. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference.
  18. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
  19. Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S Moss, and Sandra Kuebler. 2019. Monalog: a lightweight system for natural language inference based on monotonicity. arXiv preprint arXiv:1910.08772. ↗
  20. Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8713–8721. ↗
  21. Rohan Jha, Charles Lovering, and Ellie Pavlick. 2020. Does data augmentation improve generalization in nlp? arXiv preprint arXiv:2004.15012. ↗
  22. Julia Rozanova, Deborah Ferreira, Marco Valentino, Mokanrarangan Thayaparan, and Andre Freitas. 2021. Decomposing natural logic inferences in neural nli. arXiv preprint arXiv:2112.08289. ↗
  23. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. Taxinli: Taking a ride up the nlu hill. arXiv preprint arXiv:2009.14505. ↗
  24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations.
  25. Bill MacCartney. 2009. Natural language inference. Stanford University. find
  26. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Mark Steedman and Jason Baldridge. 2011. Combinatory categorial grammar. Non-Transformational Syntax: Formal and Explicit Models of Grammar. Wiley-Blackwell, pages 181–224.
  27. Robert Vacareanu, George Caique Gouveia Barbosa, Marco A. Valenzuela-Escárcega, and Mihai Surdeanu. 2020. Parsing as tagging. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5225–5231, Marseille, France. European Language Resources Association. find
  28. Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021a. Info{bert}: Improving robustness of language models from an information theoretic perspective. In International Conference on Learning Representations. find
  29. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021b. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690. ↗
  30. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  31. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40. ↗
  32. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. Help: A dataset for identifying shortcomings of neural models in monotonicity reasoning. arXiv preprint arXiv:1904.12166. ↗
  33. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. Xlnet: Generalized autoregressive pretraining for language understanding. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akoju-2023-synthetic-dataset-evaluating-complex-compositional/