Elamur Research⌕
☰
home / papers / 2025 / NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals

NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals

Paper · 2025 · Neha Srikanth and Rachel Rudinger

Summary

The paper asks how much NLI models actually understand when they label a premise–hypothesis pair. The authors split each hypothesis into small "atoms," single facts the hypothesis supports. Pairing an atom with the premise gives a small NLI sub-problem. Pairing it with the premise plus an update sentence gives a defeasible sub-problem: does the update strengthen the fact, weaken it, or leave it alone? Atoms were produced by llama-3-8b-instruct, prompted with hand-built examples modeled on Neo-Davidsonian event semantics. This was done for the δ-SNLI test set and for a 1000-example sample of the SNLI test set. On SNLI, LLMs score well on full problems, yet their atom-level answers often contradict their own overall labels. In defeasible NLI, the link between atom labels and the full label is more complex and non-monotonic. The authors define "critical atoms," the fact an example mainly tests. Grouping examples by critical atom yields a new metric, inferential consistency. These tools expose shallow understanding and reveal how repetitive benchmarks are.

Key points

  • Atom generation used llama-3-8b-instruct with exemplars in which every event and every semantic-role argument becomes its own atom. The data were δ-SNLI-TEST and SNLI-TEST-SAMPLE (1000 random SNLI test items).
  • Atoms for δ-SNLI were cleaned in two steps:
    • A DeBERTa-large model, finetuned on MNLI, Fever-NLI, Adversarial NLI, LingNLI and WANLI, filtered them. It dropped atoms the hypothesis does not entail and also atoms the premise already entails.
    • An author then hand-checked every surviving atom.
  • On SNLI the atoms were not pre-validated. Each model first judged whether H entails each atom, and consistency was scored only on the atoms that model accepted.
  • Inferential consistency is the chance that two examples sharing a critical atom are both right or both wrong. Scores ranged from 74.5 for llama-3-8b-it to 88.7 for gpt-4o-08-06. Other results: gpt-3.5-turbo 75.8, gemma-2-9b-it 76.5, gpt-4o-mini 81.7, roberta-large 82.6, llama-3-70b-it 83.5, deberta-v3-large 87.0.
  • The δ-NLI training data contain no "no effect" updates. For this reason, finetuned models were scored only on non-neutral atomic sub-problems.

Interesting ideas

  • I1. Turning a sentence into one atom for the event plus one atom per role argument gives facts that can each be checked separately against context.
  • I2. A whole-problem label should agree with the atom labels: an entailed hypothesis needs every atom entailed, and a contradiction needs at least one contradicted atom. A mismatch reveals a model that is right for the wrong reasons.
  • I3. Grouping test items by the single fact they mainly probe shows whether a fact is reliably known or only works in easy contexts. This is better than raw accuracy.
  • I4. Keeping only atoms that the hypothesis entails but the premise does not removes trivial items before costly human review.
  • I5. Finding which atom an update sentence actually targets, via the question under discussion, shows what a new piece of evidence is about.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Mova's frames already hold an event and its VerbNet roles, so each frame could be expanded into atoms that are checked one by one, explaining a verdict fact by fact.
  • I2 (answer): readers could compare their verdict on a full test question with their verdicts on its atoms and flag disagreements as suspicious answers.
  • I3 (learn): evaluation sets could be grouped by the underlying piece of knowledge, measuring whether a verb class or measure fact is handled the same way in different stories.
  • I4 (learn): silver data could be filtered the same way, using a witness model to discard items that carry no new information before LLM or human checking.
  • I5 (world): when a new sentence revises the world of a text, Mova could first find the one state or fact it bears on and adjust only that fact's plausibility.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Neha Srikanth and Rachel Rudinger. 2025. NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.130

@inproceedings{srikanth2025nli,
  title = {{NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals}},
  author = {Neha Srikanth and Rachel Rudinger},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.130},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.130}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence to discourse. ↗
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno-Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  3. Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620. ↗
  4. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.
  5. John Glover, Federico Fancellu, Vasudevan Jagannathan, Matthew R. Gormley, and Thomas Schaaf. 2022. Revisiting text decomposition methods for NLI-based factuality scoring of summaries. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 97–105, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  6. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  7. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts, USA. Association for Computational Linguistics.
  8. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. ↗
  9. OpenAI. 2024. Hello gpt-4o: A new model for openai’s future. Accessed: 2024-10-13. find
  10. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-world entailment for claims in Wikipedia. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7561–7583, Singapore. Association for Computational Linguistics.
  11. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. ↗
  12. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. ↗
  13. Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R. Bowman. 2021. Does putting a linguist in the loop improve NLU data collection? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4886–4901, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  14. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. ↗
  15. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428. ↗
  16. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  17. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439. ↗
  18. Bill MacCartney. 2009. Natural language inference. Stanford University. find
  19. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics. ↗
  20. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid Terence Parsons. 1990. Events in the semantics of english: A study in subatomic semantics. find
  21. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864–8880, Online. Association for Computational Linguistics.
  22. Raymond Reiter. 1980. A logic for default reasoning. Artificial intelligence, 13(1-2):81–132. ↗
  23. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675, Online. Association for Computational Linguistics. find
  24. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–3035. ↗
  25. Neha Srikanth, Marine Carpuat, and Rachel Rudinger. 2024a. How often are errors in natural language reasoning due to paraphrastic variability? Transactions of the Association for Computational Linguistics, 12:1143–1162. ↗
  26. Neha Srikanth and Rachel Rudinger. 2022. Partial-input baselines show that NLI models can ignore context, but they don’t. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4753–4763, Seattle, United States. Association for Computational Linguistics. ↗
  27. Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd-Graber. 2024b. Pregnant questions: The importance of pragmatic awareness in maternal health question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7253–7268, Mexico City, Mexico. Association for Computational Linguistics. ↗
  28. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, and Marek Rei. 2024. Atomic inference for NLI with generated facts as atoms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10188–10204, Miami, Florida, USA. Association for Computational Linguistics. find
  29. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. Logical reasoning with span-level predictions for interpretable and robust NLI models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3809–3823, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 880–892, Toronto, Canada. Association for Computational Linguistics. find
  30. Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, and Benjamin Van Durme. 2024. A closer look at claim decomposition. arXiv preprint arXiv:2403.11903. ↗
  31. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  32. Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. 2023. QUDeval: The evaluation of questions under discussion discourse parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5344– 5363, Singapore. Association for Computational Linguistics.
  33. Moy Yuan and Andreas Vlachos. 2024. Zero-shot fact-checking with semantic triples and knowledge graphs. In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), pages 105–115, Bangkok, Thailand. Association for Computational Linguistics. ↗
  34. Xiutian Zhao, Ke Wang, and Wei Peng. 2024. Measuring the inconsistency of large language models in preferential ranking. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 171–176. find
  35. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  36. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  37. Etsuji Tomita, Akira Tanaka, and Haruhisa Takahashi. 2006. The worst-case time complexity for generating all maximal cliques and computational experiments. Theoretical computer science, 363(1):28–42. ↗
  38. Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, and Adam Poliak. 2023. Evaluating paraphrastic robustness in textual entailment \# test examples label distr # unique hypotheses # unique generated atoms # unique valid atoms # unique atomic sub-problems % strengtheners % weakeners % no effect # unique critical atoms S NLI δ-S NLI 1000 36/32/32% (E/C/N) 1000 3263 – 3263 – – – – 1837 50/50% (S/W) 203 475 429 4079 46.0 31.4 22.6 349 *Table 5: Dataset statistics for both S NLI and δ-S NLI .* Model llama-3-8b-it gpt-3.5-turbo gemma-2-9b-it gpt-4o-mini-2024-07-18 llama-3-70b-it gpt-4o-08-06 Proportion 32.0 31.1 25.5 34.5 33.3 27.6 *Table 6: Proportion of examples in S NLI - TEST -SAMPLE (1000 examples) depending on a completeness assumption of atom generation.* Prompt A.1: Atom Generation find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srikanth-2025-nli-under-microscope-what-atomic/