Elamur Research⌕
☰
home / papers / 2025 / A MISMATCHED Benchmark for Scientific Natural Language Inference

A MISMATCHED Benchmark for Scientific Natural Language Inference

Paper · 2025 · Firoz Shaik, Mobashir Sadat, Nikita Gautam et al.

Summary

Scientific NLI asks a model to decide how two sentences taken from research papers relate: ENTAILMENT, REASONING, CONTRASTING or NEUTRAL. The two earlier resources, SciNLI and MSciNLI, draw only on computer science, so nobody knew how well models cope with other sciences. The authors built MISMATCHED, a human-annotated evaluation set with no training split. Its sentence pairs come from abstracts in PSYCHOLOGY, ENGINEERING and PUBLIC HEALTH, collected from Web of Science and NLM sources. The setup copies the mismatched-genre idea of MNLI: models train on the existing CS-only scientific NLI data and are then tested on unfamiliar fields.

Fine-tuned small models (BERT, SciBERT, RoBERTa, XLNet) and prompted LLMs (Llama-2, Llama-3, Mistral, Phi-3) all struggle. The best scores are 78.17% Macro F1 for SciBERT and 57.16% for Phi-3. The authors also mine neighbouring sentences that have no explicit connective, give them model-predicted labels and filter them. Adding these pairs helps both model families, which shows that unmarked discourse relations are a useful extra training signal.

Key points

  • MISMATCHED has 2,700 labelled pairs: a 300-pair dev set and a 2,400-pair test set, split evenly across the three domains (100 dev and 800 test pairs each).
  • Following SciNLI and MSciNLI, labels come from linking phrases such as "However" or "Therefore" at the start of the second sentence. The phrase is then deleted, and humans check the labels in dev/test.
  • The strongest baselines reach only 78.17 Macro F1 (SciBERT) and 57.16 (Phi-3), so there is large headroom.
  • About 210K implicit pairs were mined for SciNLI/MSciNLI and about 120K for MISMATCHED, roughly double the number of explicit pairs. Only pairs whose pseudo-label confidence is at least 0.6 were kept.
  • Using implicit pairs lifts SciBERT on MISMATCHED from 78.17 to 79.66. Phi-3 given implicit few-shot exemplars from MISMATCHED improves on SciNLI from 58.57 to 61.03.

Interesting ideas

  • I1. Discourse connectives can label a sentence pair for free, but the connective must be stripped afterwards, otherwise the model just learns to read the cue word.
  • I2. Model-made pseudo-labels pass two filters: a confidence threshold, and a structural check that rejects a non-neutral label on non-adjacent sentences or a neutral label on adjacent ones.
  • I3. NEUTRAL examples are made by pairing sentences that were not adjacent in the source, so negatives come almost for free with a known label.
  • I4. The benchmark contains only evaluation data from domains absent in training, much like MNLI's mismatched split, so it measures robustness directly.
  • I5. Using one in-domain example per class, taken from unlabelled text, as few-shot exemplars beat using out-of-domain gold examples.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Pairs joined by "however" or "so" could provide silver labels for inter-sentence relations in prag, with the connective removed so the relation has to be inferred from the content of the frames.
  • I2 (learn): Witness-parser silver data could likewise be dropped whenever it breaks a cheap structural invariant, which works as a built-in negative control.
  • I3 (world): Random far-apart sentence pairs could serve as "no causal/temporal link" controls when testing the event-ordering and state-tracking logic.
  • I4 (answer): Keeping test-only genres that are never used for tuning (for example, new fable collections or problem styles) would show whether the readers generalise or just fit familiar material.
  • I5 (learn): LLM teacher prompts could pick class-balanced exemplars from the same genre as the target text instead of reusing a fixed general set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States61 works · 20,107 citations🎓Kansas State UniversityManhattan · United States4 works · 190 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Firoz Shaik, Mobashir Sadat, Nikita Gautam, Doina Caragea and Cornelia Caragea. 2025. A MISMATCHED Benchmark for Scientific Natural Language Inference. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.1109

@inproceedings{shaik2025mismatched,
  title = {{A MISMATCHED Benchmark for Scientific Natural Language Inference}},
  author = {Firoz Shaik, Mobashir Sadat, Nikita Gautam, Doina Caragea and Cornelia Caragea},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.1109},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.1109}
}

References

28 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. ↗
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. find
  3. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  6. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. 8-bit Optimizers via Block-wise Quantization. arXiv preprint arXiv:2110.02861. Published as a conference paper at ICLR 2022. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. ↗
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  11. R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the Second PAS-CAL Challenges Workshop on Recognising Textual Entailment, volume 7, pages 785–794. find
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  13. Maël Jullien, Marco Valentino, and André Freitas. 2024. Semeval-2024 task 2: Safe biomedical natural language inference for clinical trials. arXiv preprint arXiv:2404.04963. ↗
  14. Maël Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Dónal Landers, and André Freitas. 2023. Nli4ct: Multi-evidence natural language inference for clinical trial reports. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16745–16764. find
  15. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  16. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. Association for Computational Linguistics. ↗
  17. Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE international conference on machine learning and applications (ICMLA), pages 364–371. IEEE.
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  19. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  20. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  21. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  22. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. ↗
  23. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276. ↗
  24. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics. ↗
  25. Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A corpus for natural language inference on scientific text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7399–7409, Dublin, Ireland. Association for Computational Linguistics. ↗
  26. Mobashir Sadat and Cornelia Caragea. 2024. MsciNLI: A diverse benchmark for scientific natural language inference. In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico. Association for Computational Linguistics. ↗
  27. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  28. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaik-2025-mismatched-benchmark-scientific-natural-language/