Elamur Research⌕
☰
home / papers / 2025 / Entailed Between the Lines: Incorporating Implication into NLI

Entailed Between the Lines: Incorporating Implication into NLI

Paper · 2025 · Shreya Havaldar, Hamidreza Alvari, John Palowitch et al.

Summary

Natural Language Inference (NLI) models are usually trained to decide whether a hypothesis follows from a premise. The authors show that this training largely ignores entailments that a reader must infer through world knowledge, pragmatics, logic or figurative language. They formalize "implied entailment" by splitting the entailment label into explicit and implied, which gives a 4-way task alongside neutral and contradiction.

They then build INLI, the Implied NLI dataset. Its pairs come from existing implicature resources, converted into premise–hypothesis form with generated alternative hypotheses for the other labels. A T5-XXL classifier trained to tell implicit from explicit entailment shows that standard benchmarks contain few implied cases. Models trained on those benchmarks perform near chance on INLI's implied entailments while staying highly accurate on explicit ones. Models fine-tuned on INLI learn the distinction and transfer it across datasets and domains.

This matters for any downstream use of NLI where unstated meaning counts, such as emotions, social cues, summarization checks and translation of figurative language.

Key points

  • INLI has 10k premises and 40k hypotheses spanning four labels: explicit entailment, implied entailment, neutral and contradiction.
  • The implied entailments are derived from two groups of sources:
    • conversational frames from LUDWIG and CIRCA (indirect answers to yes/no questions);
    • situational frames from NormBank and SocialChem (social norms and rules-of-thumb).
  • A T5-XXL implicit-vs-explicit classifier reaches 97.3% test accuracy. Annotators agreed with 92.0% of its outputs on other benchmarks (Cohen's Kappa 0.768).
  • The classifier estimates the share of implied entailments in existing benchmarks:
    • SNLI 9.33%
    • MNLI 3.68%
    • ANLI 15.66%, rising across harder rounds
    • WANLI 5.48%
  • T5-XXL trained on SNLI, MNLI or WANLI scores roughly 0.50–0.53 on INLI implied entailments, against 0.90–0.97 on explicit ones; ANLI training reaches 0.714. GPT-4 gets about 75% on INLI versus 95% on HellaSwag.

Interesting ideas

  • I1. Entailment is split by the kind of reasoning it needs: explicit when lexical semantics and syntax (synonymy, coreference, bridging) suffice, implied when an extra step such as world knowledge or pragmatics is required.
  • I2. Instead of writing implicatures from scratch, the authors convert existing structured resources (indirect QA, social-norm records) into free-form premise–hypothesis pairs, so the implied meaning comes with a trusted label.
  • I3. A classifier trained on the new label distinction is used to audit what existing benchmarks actually contain, and a human spot check confirms it.
  • I4. Scores are reported separately for explicit and implied items, which exposes chance-level behaviour that a single overall accuracy would hide.
  • I5. Generalization is tested by training on one source and testing on another, both within a domain (NormBank↔SocialChem) and across domains (situational↔conversational).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Each fact placed in the world of the text could carry a tag saying whether it came straight from the UD tree and coreference or from an inference rule or knowledge lookup. Explanations could then show which conclusions are literal and which are inferred.
  • I2 (frames): Indirect-answer and norm datasets could be recast into test pairs for the implicature and speech-act components of prag. Their reading would then be checked against labels made by people.
  • I3 (learn): A similar auditor could profile our silver data or school-test sets by how much implication they need. That would show whether training material underrepresents the hard cases.
  • I4 (answer): Results on fables, tales and school tests could be broken down by whether the answer is stated or must be inferred. This would make it clear where the reader genuinely fails.
  • I5 (learn): Repair rules and trained components could be evaluated on held-out source corpora and genres rather than random splits. This works as a negative control against memorizing one dataset's style.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya and Alex Fabrikant. 2025. Entailed Between the Lines: Incorporating Implication into NLI. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1552

@inproceedings{havaldar2025entailed,
  title = {{Entailed Between the Lines: Incorporating Implication into NLI}},
  author = {Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya and Alex Fabrikant},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1552},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1552}
}

References

46 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine translation. CoRR, abs/1707.09920. ↗
  2. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning.
  3. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. ↗
  4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. ↗
  5. U.S. Census Bureau. 2011. 2011 census. U.S. Department of Commerce. find
  6. Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021. “I’ll be there for you”: The one with understanding indirect answers. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 1–11, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics.
  7. Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620. ↗
  8. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. Rarr: Researching and revising what language models say, using language models. ↗
  9. Elizabeth Jasmi George and Radhika Mamidi. 2020. Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316–2323. ↗
  10. Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. Cider: Commonsense inference for dialogue explanation and reasoning. find
  11. Paul Grice. 1991. Studies in the Way of Words. Harvard University Press. find
  12. Siri Gullestad and Bjørn Killingmo. 2019. The theory and practice of psychoanalytic therapy: Listening for the subtext. Routledge. find
  13. Shreya Havaldar, Matthew Pressimone, Eric Wong, and Lyle Ungar. 2023. Comparing styles across languages. find
  14. Mohammad Javad Hosseini, Yang Gao, Tim Baumgärtner, Alex Fabrikant, and Reinald Kim Amplayo. 2024a. Scalable and domain-general abstractive proposition segmentation. find
  15. Mohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, and Annie Louis. 2024b. A synthetic data approach for domain generalization of nli models. arXiv preprint arXiv:2402.12368. ↗
  16. Mohammad Javad Hosseini, Filip Radlinski, Silvia Pareti, and Annie Louis. 2023. Resolving indirect referring expressions for entity selection. find
  17. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models imppressive? learning implicature and presupposition. arXiv preprint arXiv:2004.03066. ↗
  18. Donghoon Kim, Yigang Pan, and Heung Soo Park. 1998. High-versus low-context culture: A comparison of chinese, korean, and american cultures. Psychology & Marketing, 15(6):507–521. ↗
  19. Zae Myung Kim, David E. Taylor, and Dongyeop Kang. 2023. "is the pope catholic?" applying chain-of-thought reasoning to understanding conversational implicatures.
  20. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2024. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36.
  21. Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. 2022. Can language models learn from explanations in context? ↗
  22. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation. In Conference on Empirical Methods in Natural Language Processing.
  23. Annie Louis, Dan Roth, and Filip Radlinski. 2020. " i’d rather just go to bed": Understanding indirect answers. arXiv preprint arXiv:2010.03450. ↗
  24. Andrei Moldovan. 2019. Can entailments be implicatures? Philosophical Insights into Pragmatics, pages 43–62. ↗
  25. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding.
  26. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference.
  27. Zhuang Qiu, Xufeng Duan, and Zhenguang Cai. 2023. Does ChatGPT resemble humans in processing implicatures? In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 25–34, Nancy, France. Association for Computational Linguistics.
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  29. Sunny Rai, Khushang Jilesh Zaveri, Shreya Havaldar, Soumna Nema, Lyle Ungar, and Sharath Chandra Guntuku. 2024. Social norms in cinema: A cross-cultural analysis of shame, pride and prejudice. ↗
  30. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675, Online. Association for Computational Linguistics. find
  31. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2024. Can ai-generated text be reliably detected? ↗
  32. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2019. Social bias frames: Reasoning about social and power implications of language. arXiv preprint arXiv:1911.03891. ↗
  33. D. Sperber and D. Wilson. 1986. Relevance: Communication and Cognition. Language and thought series. Harvard University Press. find
  34. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI models’ performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics. ↗
  35. Junya Takayama, Tomoyuki Kajiwara, and Yuki Arase. 2021. DIRECT: Direct and indirect responses in conversational text corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1980–1989, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? ↗
  37. Fengzhu Zeng and Wei Gao. 2024. Justilm: Few-shot justification generation for explainable fact-checking of real-world claims. Transactions of the Association for Computational Linguistics, 12:334–354. ↗
  38. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal Common-sense Inference. Transactions of the Association for Computational Linguistics, 5:379–395. ↗
  39. Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, and Saab Mansour. 2024. Can your model tell a negation from an implicature? unravelling challenges with intent encoders. ↗
  40. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. Grice: A grammar-based dataset for recovering implicature and conversational reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074–2085.
  41. Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. Cobra frames: Contextual reasoning about effects and harms of offensive statements. arXiv preprint arXiv:2306.01985. ↗
  42. Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. Normbank: A knowledge bank of situational social norms. arXiv preprint arXiv:2305.17008. ↗
  43. Haoran Wang and Kai Shu. 2023. Explainable claim verification via knowledge-grounded reasoning with large language models. ↗
  44. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  45. Shisen Yue, Siyuan Song, Xinyuan Cheng, and Hai Hu. 2024a. Do large language models understand conversational implicature – a case study with a chinese sitcom.
  46. Shisen Yue, Siyuan Song, Xinyuan Cheng, and Hai Hu. 2024b. Do large language models understand conversational implicature – a case study with a chinese sitcom.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/havaldar-2025-entailed-between-lines-incorporating-implication/