Elamur Research⌕
☰
home / papers / 2026 / NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference

NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference

Summary

Earlier studies of GPT-3-era models suggested that LLMs largely ignore negation, especially in Natural Language Inference (NLI). This paper re-examines that claim for current open-source LLMs and for Portuguese, a language with few resources on the topic. Following Hossain et al. (2020), the authors distinguish "important" negations, which change the entailment label when removed, from "unimportant" ones, which leave it intact. They build NegNLI-BR from ASSIN 2 and InferBR, recast as binary NLI (Entailment vs. Non-entailment). They keep only pairs whose premise is a plain NP + VP clause with no existing negation, so a negation can be inserted and its effect on the label checked easily. The authors compare base and post-trained versions of several models on the benchmark. An input-level causal probe estimates the Average Treatment Effect of negating the premise on internal representations, with an inserted filler word as a control. Many models, including small ones, handle explicit negation well. Important negations produce a stable, significant effect on representations, and post-training usually strengthens it. The work provides a controlled Portuguese resource and a method for examining how models encode negation.

Key points

  • Data sources: NegNLI-BR is derived from ASSIN 2 (10,000 instances, labels Entailment/None) and InferBR. InferBR's Contradiction and Neutral labels were merged into Non-entailment.
  • Filtering: The Stanza constituent parser (Qi et al., 2020) was used to keep premises with a simple Noun Phrase + Verb Phrase structure. Premises already containing a simple negation were discarded.
  • Design goals: The dataset was built so that negation interventions are easy to apply, their effect on the label is easy to verify, and models cannot rely on classification shortcuts.
  • Research questions: RQ1 asks how post-training affects task performance (base vs. post-trained models). RQ2 asks whether important negations are encoded in representations, measured via a causal probe in Pearl's framework.
  • Findings:
    • The Average Treatment Effect of important negation is stable for most tested models and differs from that of unimportant negations and filler words.
    • Post-trained models are, on average, more sensitive to important negation than their base counterparts.
    • Per-model accuracy figures were not included in the provided text extract.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3mrey394c68a

@inproceedings{westhelle2026negnli,
  title = {{NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference}},
  author = {Westhelle, Matheus and Moreira, Viviane},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3mrey394c68a},
  url = {https://doi.org/10.63317/3mrey394c68a}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Amini, Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2023. Naturalistic Causal Probing for Morpho-Syntax. Transactions of the Association for Computational Linguistics, 11:384– 403. find
  2. Luciana Bencke, Francielle Vasconcellos Pereira, Moniele Kunrath Santos, and Viviane Moreira. 2024. InferBR: A natural language inference dataset in Portuguese. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024… find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  5. Nicholas Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2024. Tucano: Advancing neural text generation for portuguese. find
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. find
  9. Daya others Guo. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  10. Laurence R. Horn. 2001. A Natural History of Negation. The David Hume Series. CSLI. find
  11. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106… find our card
  12. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720. find our card
  13. Aishwarya Kamath, Johan Ferret, Shreya Pathak, et al. 2025. Gemma 3 technical report. ArXiv, abs/2503.19786. find
  14. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics. find our card
  15. Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, and Jie Tang. 2025. A survey of post-training scaling in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Comput… find
  16. Mistral AI. 2024. Mistral AI Models on Hugging Face. Models: 12B Nemo, 22B Small, 123B Large. find
  17. Andrew Moore and Jeremy Barnes. 2021. Multi-task learning of negation and speculation for targeted sentiment classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2838–2869, Online. Ass… find
  18. Roser Morante and Walter Daelemans. 2012. Conandoyle-neg: Annotation of negation cues and their scope in conan doyle stories. In International Conference on Language Resources and Evaluation. find
  19. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish I… find
  20. OpenAI, Josh Achiam, Steven Adler, et al. 2024. Gpt-4 technical report. find our card
  21. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  22. Judea Pearl. 2009. Causal inference in statistics: An overview. Statistics Surveys, 3(none):96 – 146. find
  23. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  24. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.… find our card
  25. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194–209, Online. Association… find
  26. Livy Real, Erick Fonseca, and Hugo Gonçalo Oliveira. 2020. The assin 2 shared task: A quick overview. In Computational Processing of the Portuguese Language: 14th International Conference, PROPOR 2020, Evora, Portugal, March 2–4, 2020, Proceedings, page 406–412, Berlin, Heidelberg. Springer-Verlag. find
  27. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8:842–866. find our card
  28. PAUL R. ROSENBAUM and DONALD B. RU-BIN. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55. find
  29. Paul R. Rosenbaum and Donald B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55. find
  30. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. ArXiv, abs/2408.03314. find our card
  31. Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023. Language models are not naysayers: An analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 101–114, Toronto, Canada. Associa… find
  32. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. find
  33. Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The bioscope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics, 9(11):S9. find
  34. Tereza Vrabcová, Marek Kadlčík, Petr Sojka, Michal Štefánik, and Michal Spiegel. 2025. Negation: A Pink Elephant in the Large Language Models’ Room? find
  35. Orion Weller, Dawn Lawrie, and Benjamin Van Durme. 2024. NevIR: Negation in neural information retrieval. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2274–2287, St. Julian’s, Malta. Association for Comp… find
  36. An Yang et al. 2025a. Qwen2.5 technical report. find
  37. An Yang et al. 2025b. Qwen3 technical report. arXiv preprint arXiv:2505.09388. find our card
  38. Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z. HaoChen, James Zou, Percy Liang, and Serena Yeung. 2023. Beyond positive scaling: How negation impacts scaling trends of language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7479–7498, Toronto, Canada.… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/westhelle-2026-negnli-br-brazilian-portuguese-benchmark/