Elamur Research⌕
☰
home / papers / 2026 / Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials

Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials

Paper · 2026 · Mathilde Aguiar, Pierre Zweigenbaum and Nona Naderi

Summary

LLMs do well on natural language inference over clinical trial reports, but it is unclear which kinds of reasoning they handle and which they fail. The authors add a reasoning layer to the test set of NLI4CT, a corpus of English breast-cancer trial reports from clinicaltrials.gov in which statements must be judged as entailed or contradicted by report sections. They build a scheme of 17 fine-grained inference types grouped into six super-classes: Task-related expression, Logical, Generic knowledge, Biomedical, Numerical, and Typo/Error. Each of the 500 statement–premise pairs gets every type needed to reach the verdict. They then prompt nine LLMs, both decoder-only and sequence-to-sequence, from general and medical domains, and break down performance by inference type. Logical reasoning lowers results for several Qwen models, and numerical reasoning lowers results for Flan-T5-XL and Mixtral. The number of types involved and the premise section also matter. The resource lets researchers diagnose model weaknesses at the level of reasoning skill rather than through a single accuracy score.

Key points

  • The annotations total 1,949 labels over the 500 test pairs of NLI4CT, using 17 types in 6 super-classes. Code and data are released on GitHub.
  • The guidelines were built step by step:
    • a first pilot subset of 50 instances (10%), stratified by Entailment/Contradiction and Single/Comparison;
    • revision of the guidelines;
    • an independent round on 50 more instances by three annotators.
  • Agreement on that round was high. Pairwise Krippendorff's α averaged 0.93, and the per-type average was 0.98. The remaining pairs were labelled by one annotator.
  • Re-annotating the NLI labels gave α = 0.97 against the originals and uncovered a few gold errors. The corrected split is 253 Contradiction and 247 Entailment.
  • Premises average 125 words, with a maximum of 1,388 words. Statements average 22 words.
  • Logical inferences hurt Qwen3-4B, Qwen2.5-7B and Qwen2.5-14B. Numerical inferences hurt Flan-T5-XL and Mixtral. MMed-Llama-3 has trouble with the structure of trial reports.

Interesting ideas

  • I1. Tag each test item with all the reasoning skills it requires, then compute accuracy per skill; this shows where a system breaks instead of hiding failures in one average.
  • I2. Logical connectives are treated as separate inference types: conjunction, disjunction, negation and contrast. These small words often decide the verdict, so they deserve their own diagnostics.
  • I3. Grow the label scheme from a small stratified pilot sample, revise the guidelines, then measure agreement on a fresh independent batch. This yields a taxonomy that covers the data without overlapping labels.
  • I4. While adding the new layer, annotators re-judged the original gold labels and found mistakes. A second annotation pass doubles as a cheap audit of the benchmark itself.
  • I5. Performance is also sliced by how many inference types co-occur and by which document section the premise comes from. Difficulty grows with the number of hops and with the kind of text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Tag school-test and fable QA items with the skills they need, such as negation, arithmetic or coreference, so we can see which reader stage loses points.
  • I2 (frames): Make scope of negation, coordination and contrast an explicit, checkable part of frames, and evaluate it on its own slice.
  • I3 (learn): Use the same loop when defining error categories for self-induced repair rules or LLM-teacher outputs, and confirm agreement before scaling up.
  • I4 (learn): When building evaluation sets, re-verify gold answers as a negative control, so that "errors" caused by bad gold are not counted against the system.
  • I5 (world): Report answer accuracy by the number of world-state steps needed and by text genre, to find where the event and entity model stops scaling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksReasoning · 1,426 works

Cite

Cite as Mathilde Aguiar, Pierre Zweigenbaum and Nona Naderi. 2026. Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/359toazp33g8

@inproceedings{aguiar2026assessing,
  title = {{Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials}},
  author = {Mathilde Aguiar, Pierre Zweigenbaum and Nona Naderi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/359toazp33g8},
  url = {https://doi.org/10.63317/359toazp33g8}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Agresti. 2013. Categorical data analysis. John Wiley & Sons. find
  2. Hyung Won Chung et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  3. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The Llama 3 herd of models. CoRR, abs/2407.21783. ↗
  6. Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard. ↗
  7. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  8. Albert Q. Jiang et al. 2023. Mistral 7b. ArXiv, abs/2310.06825. ↗
  9. Albert Q. Jiang et al. 2024. Mixtral of experts. ArXiv, abs/2401.04088. ↗
  10. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41–55, Online. Association for Computational Linguistics. find
  11. Mael Jullien, Marco Valentino, and André Freitas. 2024. SemEval-2024 task 2: Safe biomedical natural language inference for clinical trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1947–1962, Mexico City, Mexico. Association for Computational Linguistics. ↗
  12. Mael Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Dónal Landers, and Andre Freitas. 2023a. NLI4CT: Multi-evidence natural language inference for clinical trial reports. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16745–16764, Singapore. Association for Computational Linguistics. find
  13. Maël Jullien, Marco Valentino, Hannah Frost, Paul O’regan, Donal Landers, and André Freitas. 2023b. SemEval-2023 task 7: Multi-evidence natural language inference for clinical trial data. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2216–2226, Toronto, Canada. Association for Computational Linguistics. find
  14. Mael Jullien, Marco Valentino, Leonardo Ranaldi, and Andre Freitas. 2025. Dissecting clinical reasoning in language models: A comparative study of prompts and model adaptation strategies. ↗
  15. Kamal Raj Kanakarajan and Malaikannan Sankarasubbu. 2023. Saama AI research at SemEval-2023 task 7: Exploring the capabilities of flan-t5 for multi-evidence natural language inference in clinical trial data. In SemEval-2023, pages 995– 1003, Toronto, Canada. find
  16. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. ↗
  17. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. ↗
  18. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  19. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1). ↗
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  21. Adina Williams, Tristan Thrush, and Douwe Kiela. 2022. ANLIzing the adversarial natural language inference dataset. In Proceedings of the Society for Computation in Linguistics 2022, pages 23–54, online. Association for Computational Linguistics.
  22. Liyun Yan, Danni E, Mei Gan, Cyril Grouin, and Mathieu Valette. 2020. Inference annotation of a Chinese corpus for opinion mining. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4991–4999, Marseille, France. European Language Resources Association. find
  23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 technical report. ↗
  24. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aguiar-2026-assessing-difficulty-inference-types-natural/