Elamur Research⌕
☰
home / papers / 2023 / Curing the SICK and Other NLI Maladies

Curing the SICK and Other NLI Maladies

Paper · 2023 · Aikaterini-Lida Kalouli, Hai Hu, Alexander F. Webb et al.

Summary

The paper argues that much of the noise in NLI datasets comes from a hidden conflict between two kinds of judgment. Annotators mix strict logical inference with everyday common-sense inference, and this matters more than general uncertainty. Using SICK as a case study, the authors first label large parts of the corpus automatically with symbolic inference systems (ccg2lambda, LangPro, MonaLog, GKR4NLI). They then hand-annotate the rest under detailed semantic guidelines covering coreference, aspect and vagueness. Even very specific guidelines leave real disagreement. They therefore propose an "ordered sense space":

  • one end holds impossible (non-sensical) inferences;
  • the other end holds strictly logical ones;
  • the middle is a graded common-sense continuum where labels can legitimately differ.

Each pair receives a logic label and a common-sense label. Fine-tuning pre-trained language models on the re-annotated SICK shows marked performance differences between the two views. A small crowd study on 100 MultiNLI pairs taken from chaosNLI suggests that non-experts can use the scheme. The work offers a cleaner, explainable route to reliable NLI data.

Key points

  • The entire SICK corpus is re-annotated. The refined data and the 100 MultiNLI crowd annotations are released at github.com/huhailinguist/curing-SICK.
  • The test set is split into SICK.test.logic and SICK.test.common-sense, and systems are scored separately on each.
  • Symbolic systems are very precise on contradiction and entailment (roughly 97–99% precision in most settings) but miss many cases (recall often 40–65%). This makes them trustworthy partial annotators.
  • Weighted F1 on the original SICK.test:
    • ccg2lambda: about 80.6
    • LangPro: 80.4
    • GKR4NLI: 76.9
    • MonaLog: 73.4
  • On the logic split, GKR4NLI scores best (78.9).
  • The appendix gives a taxonomy of annotation troubles, each with a prototypical example: idioms, looseness of word meaning, antonymous subjects or predicates, gender/age terms, privative adjectives, word senses, structural ambiguity, world knowledge, and coreference.

Interesting ideas

  • I1. Give each inference two separate verdicts, one strictly logical and one "person on the street", instead of forcing a single compromise label. Disagreement then becomes signal rather than noise.
  • I2. Let high-precision, low-recall rule-based systems label what they can, and send only the leftover cases to human (or teacher) annotation.
  • I3. Before calling a pair contradictory, first try to make the subject or the event coreferent; pairs with nothing linkable are treated as neutral, not contradictory.
  • I4. Catalogue recurring sources of disagreement (privative adjectives like "fake" or "animated", loose near-synonyms like big/huge, gender-versus-age nouns) with one prototypical example each.
  • I5. Evaluate on separate test splits per inference regime, so a model's strength on logic is not hidden by its behaviour on common sense, or the reverse.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): readers could return a strict answer and a default common-sense answer side by side, each with its own justification chain.
  • I2 (learn): silver data could keep only items where precise symbolic witnesses fire, and route the rest to LLM teacher tiers.
  • I3 (frames): coreference across frames could be resolved by this explicit alignment step before contrastive reasoning, which keeps contradiction decisions traceable.
  • I4 (world): the ontology could mark privative modifiers and loose synonym pairs so the world model knows when an entity claim is blocked or only default-true.
  • I5 (learn): evaluation suites could report logic-only and common-sense-only scores to show which reasoning layer regressed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations🎓Indiana UniversityBloomington · United States41 works · 3,663 citations🤝Topos InstituteBerkeley · United States4 works · 19 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Aikaterini-Lida Kalouli, Hai Hu, Alexander F. Webb, Lawrence S. Moss and Valeria de Paiva. 2023. Curing the SICK and Other NLI Maladies. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00465

@article{kalouli2023curing,
  title = {{Curing the SICK and Other NLI Maladies}},
  author = {Aikaterini-Lida Kalouli, Hai Hu, Alexander F. Webb, Lawrence S. Moss and Valeria {de Paiva}},
  journal = {Computational Linguistics},
  year = {2023},
  doi = {10.1162/coli\_a\_00465},
  url = {https://doi.org/10.1162/coli_a_00465}
}

References

76 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abzianidze, Lasha. 2014. Towards a wide-coverage tableau method for natural logic. In New Frontiers in Artificial Intelligence - JSAI-isAI 2014 Workshops, LENLS, JURISIN, and GABA, Kanagawa, Japan, October 27–28, 2014, Revised Selected Papers, pages 66–82. https://doi.org/10.1007/978-3-662-48119-6 6 ↗
  2. Abzianidze, Lasha. 2015. A tableau prover for natural logic and language. In Proceedings of EMNLP, pages 2492–2502. https://doi.org/10.18653/v1/D15-1296 ccg2lambda12 LangPro MonaLog GKR4NLI 98.9 | 64.0 96.9 | 54.2 75.4 | 99.0 79.9 97.3 | 65.5 97.4 | 55.2 76.0 | 99.0 80.4 81.2 | 64.5 97.8 | 41.1 71.2 | 95.7 73.4 74.6 | 80.3 76.5 | 60.5 78.3 | 85.0 76.9 99.8 | 44.2 98.5 | 59.7 69.4 | 99.6 75.4 98.5 | 45.2 97.4 | 59.8 69.8 | 99.3 75.6 98.9 | 53.5 99.3 | 45.2 67.7 | 99.6 73.3 89.4 | 65.2 77.6 | 65.5 77.6 | 92.6 78.9 99.6 | 43.2 98.2 | 56.6 67.0 | 99.5 73.4 98.1 | 44.2 97.9 | 57.1 67.4 | 99.5 73.8 99.1 | 52.6 99.0 | 42.8 65.4 | 99.6 71.3 90.0 | 64.4 78.9 | 63.2 75.4 | 93.2 77.9 ↗
  3. Abzianidze, Lasha. 2016. A Natural Proof System for Natural Language. Ph.D. thesis, Tilburg University. find
  4. Abzianidze, Lasha. 2017. LangPro: Natural language theorem prover. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 115–120. https://doi.org/10.18653/v1/D17-2020 ↗
  5. Bar-Haim, Roy, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Recognising Textual Entailment Challenge Workshop, pages 1–9. find
  6. Bentivogli, Luisa, Peter Clark, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2010. The sixth PASCAL recognizing textual entailment challenge. In Proceedings of the 10th Text Analysis Conference (TAC), 18 pages. find
  7. Bentivogli, Luisa, Peter Clark, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2011. The seventh PASCAL recognizing textual entailment challenge. In Proceedings of the 11th Text Analysis Conference (TAC), 16 pages. find
  8. Bentivogli, Luisa, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the 9th Text Analysis Conference (TAC), 15 pages. find
  9. Bhagavatula, Chandra, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, 18 pages. ↗
  10. Bobrow, Daniel G., Bob Cheslow, Cleo Condoravdi, Lauri Karttunen, Tracy Holloway King, Rowan Nairn, Valeria, de Paiva, Charlotte Price, and Annie, Zaenen. 2007. PARC’s bridge and question answering system. In Proceedings of the Grammar Engineering Across Frameworks Workshop (GEAF 2007) pages 46–66. find
  11. Bowman, Samuel R., Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. ↗
  12. Camburu, Oana Maria, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with natural language explanations. In NeurIPS, pages 9539–9549. ↗
  13. Chen, Tongfei, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain Natural Language Inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779.
  14. Chierchia, G. and S. McConnell-Ginet. 2001. Meaning and Grammar: An Introduction to Semantics, 2 edition. MIT Press. find
  15. Clark, Stephen and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493–552. https://doi.org/10.1162/coli .2007.33.4.493 ↗
  16. Cooper, Robin, Stergios Chatzikyriakidis, and Simon Dobnik. 2016. Testing the FraCaS test suite. In Proceedings of the Logic and Engineering of Natural Language Semantics Workshop. find
  17. Crouch, Richard, Lauri Karttunen, and Annie Zaenen. 2006. Circumscribing is not excluding: A response to Manning. Unpublished manuscript. http://www2.parc.com/istl/members/karttune/publications/reply-tomanning.pdf. ↗
  18. Crouch, Richard and Tracy Holloway King. 2007. Systems and methods for detecting entailment and contradiction. US Patent 7,313,515. find
  19. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the Machine Learning Challenges Workshop, pages 177–190. find
  20. Dasgupta, Ishita, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating compositionality in sentence embeddings. CoRR, abs/1802.04302. ↗
  21. de Marneffe, Marie Catherine, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039–1047.
  22. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  23. Fitting, Melvin. 1990. First-order Logic and Automated Theorem Proving. Springer-Verlag. ↗
  24. Geva, Mor, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166. https://doi.org/10 .18653/v1/D19-1107 ↗
  25. Giampiccolo, Danilo, Hoa Trang Dang, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2008. The fourth PASCAL recognizing textual entailment challenge. In Proceedings of the 8th Text Analysis Conference (TAC), 8 pages. https://doi.org/10.3115/1654536.1654538 ↗
  26. Giampiccolo, Danilo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the Workshop on Textual Entailment and Paraphrasing, pages 1–9. https://doi.org/10.3115/1654536.1654538 ↗
  27. Glickman, Oren, Ido Dagan, and Moshe Koppel. 2005. A probabilistic classification approach for lexical textual entailment. Proceedings of the National Conference on Artificial Intelligence, 3:1050–1055. find
  28. Glockner, Max, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. https://doi.org/10 .18653/v1/P18-2103 ↗
  29. Gorman, Kyle and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791. find
  30. Gururangan, Suchin, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in Natural Language Inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. https://doi.org/10.18653/v1/N18 -2017 ↗
  31. Horn, Laurence. 1989. A Natural History of Negation. University of Chicago Press. find
  32. Hu, Hai, Qi Chen, and Larry Moss. 2019. Natural Language Inference with monotonicity. In Proceedings of the 13th International Conference on Computational Semantics - Short Papers, pages 8–15.
  33. Hu, Hai, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S. Moss, and Sandra Kuebler. 2020. MonaLog: A lightweight system for Natural Language Inference based on monotonicity. In Proceedings of SCiL, pages 334–344. ↗
  34. Hu, Hai and Lawrence S. Moss. 2018. Polarity computations in flexible categorial grammar. In Proceedings of *SEM, pages 124–129. https://doi.org/10 .18653/v1/S18-2015 ↗
  35. Kalouli, Aikaterini Lida. 2021. Hy-NLI: A Hybrid system for state-of-the-art Natural Language Inference. Ph.D. thesis, Universität Konstanz, Konstanz. find
  36. Kalouli, Aikaterini Lida, Annebeth Buis, Livy Real, Martha Palmer, and Valeria dePaiva. 2019. Explaining simple Natural Language Inference. In Proceedings of the 13th Linguistic Annotation Workshop, pages 132–143. find
  37. Kalouli, Aikaterini Lida and Richard Crouch. 2018. GKR: The Graphical Knowledge Representation for semantic parsing. In Proceedings of the Workshop on Computational Semantics beyond Events and Roles, pages 27–37. https://doi.org/10.18653/v1/W18-1304 ↗
  38. Kalouli, Aikaterini Lida, Richard Crouch, and Valeria de Paiva. 2020. Hy-NLI: A hybrid system for Natural Language Inference. In Proceedings of the 28th International Conference on Computational Linguistics, COLING ’20, pages 5235–5249. https://doi.org/10.18653/v1/2020 .coling-main.459 ↗
  39. Kalouli, Aikaterini Lida, Livy Real, Annebeth Buis, Martha Palmer, and Valeria de Paiva. 2021. Annotation difficulties in Natural Language Inference. In Proceedings of the Symposium in Information and Human Language Technology, pages 247–254. https://doi.org/10.5753/stil.2021.17804 ↗
  40. Kalouli, Aikaterini Lida, Livy Real, and Valeria de Paiva. 2017a. Correcting contradictions. In Proceedings of Computing Natural Language Inference (CONLI) Workshop, 6 pages. find
  41. Kalouli, Aikaterini Lida, Livy Real, and Valeria de Paiva. 2017b. Textual inference: Getting logic from humans. In Proceedings of the 12th International Conference on Computational Semantics (IWCS), 7 pages. find
  42. Kalouli, Aikaterini Lida, Livy Real, and Valeria de Paiva. 2018. WordNet for “easy” textual inferences. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 35–42. https://doi.org/10 .1609/aaai.v32i1.12022 ↗
  43. Khot, Tushar, Ashish Sabharwal, and Peter Clark. 2018. SciTaiL: A textual entailment dataset from science question answering. In AAAI, pages 5189–5197. find
  44. Lewis, Mike and Mark Steedman. 2014. A* CCG parsing with a supertag-factored model. In Proceedings of EMNLP, pages 990–1000. https://doi.org/10 .3115/v1/D14-1107 ↗
  45. Liu, Xiaodong, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496.
  46. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  47. MacCartney, Bill. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. AAI3364139. find
  48. Manning, Christopher D. 2006. Local textual inference: It’s hard to circumscribe, but you know it when you see it—and NLP needs it. https://nlp.stanford.edu/manning/papers/TextualInference .pdf. ↗
  49. Marelli, Marco, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 1–8. https://doi.org/10.3115/v1/S14-2001 ↗
  50. Marelli, Marco, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223.
  51. Mates, Benson. 1972. Elementary Logic. Oxford University Press. find
  52. McCoy, Tom, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. https://doi.org/10.18653/v1/P19 -1334 ↗
  53. Naik, Aakanksha, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. ↗
  54. Nie, Yixin, Yicheng Wang, and Mohit Bansal. 2018. Analyzing compositionality-sensitivity of NLI models. CoRR, abs/1811.07033:6867–6874. https://doi.org/10.1609/aaai.v33i01 .33016867 ↗
  55. Nie, Yixin, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of ACL, pages 4885–4901. https://doi.org/10 .18653/v1/2020.acl-main.441 ↗
  56. Nie, Yixin, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on Natural Language Inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143. https://doi.org/10 .18653/v1/2020.emnlp-main.734 ↗
  57. Partee, Barbara H. 2010. Privative adjectives: Subsective plus coercion. Presuppositions and Discourse: Essays Offered to Hans Kamp. Brill, pages 273–285. https://doi.org/10.1163/9789004253162 011 ↗
  58. Pavlick, Ellie and Chris Callison-Burch. 2016. Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173. https://doi.org/10.18653/v1/P16 -1204 ↗
  59. Pavlick, Ellie and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  60. Pilault, Jonathan, Amine Elhattami, and Christopher Pal. 2020. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. arXiv preprint arXiv:2009.09139v1. ↗
  61. Poesio, Massimo and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83. https://doi.org/10 .3115/1608829.1608840 ↗
  62. Poliak, Adam, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. https://doi.org/10 .18653/v1/S18-2023 ↗
  63. Prabhakaran, Vinodkumar, Aida Mostafazadeh Davani, and Mark Dı́az. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138. https://doi.org/10 .18653/v1/2021.law-1.14 ↗
  64. Richardson, Kyle, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2020. Probing Natural Language Inference models through semantic fragments. In the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, the Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, the Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, pages 8713–8721. https://doi.org/10 .1609/aaai.v34i05.6397 ↗
  65. Sánchez-Valencia, Victor. 1991. Studies on Natural Logic and Categorial Grammar. Ph.D. thesis, University of Amsterdam. find
  66. van Benthem, Johan. 2008. A brief history of natural logic. In M. Chakraborty, B. Lowe, M. Nath Mitra, and S. Sarukki, editors, Logic, Navya-Nyaya and Applications: Homage to Bimal Matilal. College Publications. find
  67. van der Goot, Rob. 2021. We need to talk about train-dev-test splits. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4485–4494. https://doi.org/10 .18653/v1/2021.emnlp-main.368 ↗
  68. Versley, Yannick. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6:333–353. ↗
  69. Williams, Adina, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. https://doi.org/10 .18653/v1/N18-1101 ↗
  70. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. https://doi.org/10.18653/v1/2020.emnlp -demos.6 ↗
  71. Yanaka, Hitomi, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6105–6117. https://doi.org/10.18653/v1/2020.acl-main .543 ↗
  72. Yanaka, Hitomi, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of BlackboxNLP, pages 31–40. https://doi.org/10.18653/v1/W19-4804 ↗
  73. Yanaka, Hitomi, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. Help: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of *SEM, pages 250–255. https://doi.org/10.18653/v1/S19-1027 ↗
  74. Yanaka, Hitomi, Koji Mineshima, Pascual Martı́nez-Gómez, and Daisuke Bekki. 2018. Acquisition of phrase correspondences using natural deduction proofs. In Proceedings of NAACL, pages 756–766. ↗
  75. Zaenen, Annie, Lauri Karttunen, and Richard Crouch. 2005. Local textual inference: Can it be defined or circumscribed? In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 31–36. https://doi.org/10.3115/1631862.1631868 ↗
  76. Zhang, Sheng, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. https://doi.org/10.1162/tacl a 00068 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kalouli-2023-curing-sick-other-nli-maladies/