Elamur Research⌕
☰
home / papers / 2025 / Math Natural Language Inference: this should be easy!

Math Natural Language Inference: this should be easy!

Paper · 2025 · Valeria de Paiva, Qiyue Gao, Hai Hu et al.

Summary

The paper tests whether current LLMs can handle natural language inference on research-level mathematical prose, a task the authors call Math NLI, and whether LLMs can build and label such corpora themselves. Premises come from abstracts in the TAC (Theory and Applications of Categories) corpus. Researchers with both mathematical and NLI experience wrote entailed, contradictory and neutral hypotheses for 31 selected sentences, giving a seed set of 279 pairs. A second corpus has GPT-written hypotheses. Ten LLMs labelled both sets. The authors measured accuracy and also how often the models agreed with one another. Items where models clashed with the human labels were analysed by hand to find recurring error types. Accuracy on the human set is moderate to high. Neutral pairs are the main weak spot, and models still slip on simple inferences. In some settings, a majority vote of LLMs is roughly as good as human labels. Hypothesis-only shortcuts no longer seem to work for these models. The authors conclude that mathematical text does not make NLI easier, and they release both corpora.

Key points

  • Seed corpus: 31 premises drawn from the 432 "Goldilocks" sentences (moderate length, little LaTeX) in the TAC corpus of about 3K sentences. Three annotators each wrote one E, one C and one N hypothesis per premise, giving 279 balanced pairs.
  • Ten LLMs were evaluated in two groups:
    • Group 1: GPT-4, Llama 2, Llama 3, Claude 3, Mistral-large.
    • Group 2: Llama-3.1-70B-Instruct, Qwen2-72B-Instruct, Mixtral-8x22B, deepseek-llm-67b-chat, gemma-2-27b-it.
  • Accuracy on the seed corpus ranged from 71% to 91%. Most models labelled E and C pairs well but misjudged N pairs.
    • Llama 3 got only 48% of neutral pairs right and called 35% of them contradictions.
    • Claude 3 did best on N, at 84.9%.
  • About 20 "red pairs" contradicted human labels. Their failure types were:
    • ignored context;
    • vague quantifiers;
    • lexical/pragmatic ambiguity, such as "resemble";
    • naming of math entities;
    • unknown math concepts.
  • On a hypothesis-only baseline, the Group 2 models labelled nearly every item N (0.96–1.00), which is the correct behaviour. On the GPT-generated corpus, accuracy ranged from about 60% to 87%. Both corpora are released at github.com/MathNLI/MathNLI.

Interesting ideas

  • I1. Pairs where 4–5 independent models disagree with the gold label are pulled aside as "red pairs" for expert review, and they are discarded if experts find them controversial; this turns model disagreement into a cheap way to find bad or ambiguous data.
  • I2. A hypothesis-only run, where the model sees no premise, should yield nearly all N, so it works as a negative control for shortcut learning.
  • I3. Annotators treated domain concepts as black boxes, writing inferences that need no knowledge of what a term means; this works because coined names can mislead (for instance, skew monoidal categories do not in general qualify as monoidal ones).
  • I4. A common model error was dropping a scoping condition such as "in the nilpotent case" and comparing only the main clauses, which turned a neutral pair into a contradiction.
  • I5. Some words carry a defeasible implicature: "resemble" suggests the two things are not identical, and "numerous examples" was judged by experts to entail "two examples".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data agreement could apply the same rule. Sentences where most witness parsers contradict the current gold get routed to review rather than trained on.
  • I2 (learn): Mova's readers could be fed questions with the story removed. Any answer above chance would signal reliance on surface cues instead of the built world.
  • I3 (world): Unknown compound terms should become opaque entities with no assumed hypernym from head-word matching. Then "V-n-category" would not silently inherit from "V-category".
  • I4 (frames): Frames should keep conditional and restrictive adjuncts as part of the asserted proposition. Contradiction checks should then fire only when the scopes match.
  • I5 (frames): The implicature module could hold a small lexicon of such cancellable inferences, marked separately from strict entailments. That would let answers explain which reading they relied on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓University of California San DiegoSan Diego · United States73 works · 51,948 citations🎓Indiana UniversityBloomington · United States41 works · 3,663 citations🤝Topos InstituteBerkeley · United States4 works · 19 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Valeria de Paiva, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S. Moss and Zhiheng Qian. 2025. Math Natural Language Inference: this should be easy!. Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.starsem-1.14

@inproceedings{paiva2025math,
  title = {{Math Natural Language Inference: this should be easy!}},
  author = {Valeria {de Paiva}, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S. Moss and Zhiheng Qian},
  booktitle = {Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.starsem-1.14},
  url = {https://doi.org/10.18653/v1/2025.starsem-1.14}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗
  2. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. ↗
  3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. ↗
  4. Jacob Collard, Valeria de Paiva, Brendan Fong, and Eswaran Subrahmanian. 2022. Extracting mathematical concepts from text. Preprint, arXiv:2208.13830. ↗
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  6. Wesley H. Holliday, Matthew Mandelkern, and Cedegao E. Zhang. 2024. Conditional and modal reasoning in large language models. Preprint, arXiv:2401.17169. ↗
  7. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. ↗
  8. Aikaterini-Lida Kalouli, Valeria de Paiva, and Livy Real. 2017. Correcting contradictions. In Proceedings of the Computing Natural Language Inference Workshop. find
  9. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2024. Causal reasoning and large language models: Opening a new frontier for causality. Preprint, arXiv:2305.00050. ↗
  10. Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, and Dieuwke Hupkes. 2024. Lost in inference: Rediscovering the role of natural language inference for large language models. arXiv preprint arXiv:2411.14103. ↗
  11. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  12. Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, and Barbara Plank. 2024. To know or not to know? analyzing self-consistency of large language models under ambiguity. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17203–17217, Miami, Florida, USA. Association for Computational Linguistics. find
  13. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. 4. P: We define eventually cyclic Boolean flows and the eventually cyclic spectrum of a Boolean flow. H: The definition of the eventually cyclic spectrum of a Boolean flow uses the definition of eventually cyclic Boolean flows. ↗
  14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  15. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625:476 – 482. 5. P: The axioms resemble those for monoidal Abelian categories with the addition of an involutive functor. H: The axioms are the ones of monoidal Abelian categories. ↗
  16. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 6. P: The category of Set-valued presheaves on a small category B is a topos. H: The category of Set-valued presheaves on a small category C is a topos.
  17. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2024. Are large language models really good logical reasoners? a comprehensive evaluation and beyond. Preprint, arXiv:2306.09841. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paiva-2025-math-natural-language-inference-this/