Elamur Research⌕
☰
home / papers / 2025 / When Does Meaning Backfire? Investigating the Role of AMRs in NLI

When Does Meaning Backfire? Investigating the Role of AMRs in NLI

Paper · 2025 · Junghyun Min, Xiulin Yang and Shira Wein

Summary

The paper asks whether explicit semantic structure, in the form of Abstract Meaning Representation (AMR) graphs, helps pretrained models handle Natural Language Inference (NLI) more robustly than shallow lexical cues do. The authors parse every sentence in MNLI and HANS with an off-the-shelf amrlib parser (BART-large trained on AMR 3.0). They then test two settings. In the first, they fine-tune bert-base-uncased with AMR added as linearized text, as a graph-neural-network embedding, or as the only input. In the second, they prompt GPT-4o in zero-shot and 5-shot modes with the sentence alone, the AMR alone, or both together. In fine-tuning, AMR brings no benefit and slightly lowers HANS accuracy. In prompting, sentence plus AMR improves GPT-4o on HANS but hurts it on MNLI. An ablation and a regression analysis show that the HANS gain does not come from better semantic reasoning. AMR makes small surface differences look larger, which pushes the model toward answering non-entailment. This is a useful caution for anyone who expects symbolic meaning graphs to remove heuristic behaviour automatically.

Key points

  • Data: MNLI has 297k training pairs and is evaluated on the matched dev set (about 10k pairs). HANS has 30k template-generated pairs with two labels, so neutral and contradiction predictions are merged into non-entailment.
  • Fine-tuning used BERT-base for 3 epochs at learning rate 2e-5, averaged over 10 seeds, with MNLI/HANS accuracy as follows:
    • text-only baseline: 0.84/0.52
    • +AMR as text: 0.83/0.47
    • +AMR as graph: 0.84/0.49
    • AMR as text only: 0.74/0.51
  • A fine-tuned model given only AMR labels 98.3% of HANS items as entailment, compared with 94.7% for the text baseline. The overlap heuristic therefore becomes stronger, not weaker.
  • Zero-shot GPT-4o, MNLI/HANS accuracy:
    • text only: 0.91/0.82
    • +AMR: 0.75/0.87
    • AMR only: 0.68/0.70
  • A logistic regression with SMATCH++ similarity between premise and hypothesis AMRs found an interaction term that was negative and statistically significant (β = -0.042, p < 2e-16). Graph similarity predicts the model's outputs better than the gold labels do.
  • Adding AMR gives no robustness to longer inputs on MNLI. Sentence-only prompts stay ahead in most length bins.

Interesting ideas

  • I1. Regress a model's predictions on a structural-overlap score and test whether that score explains the outputs better than the gold labels do; this exposes "matching on form" without manual error analysis.
  • I2. Run three matched input conditions (text only, structure only, text plus structure) so that any gain can be credited to the right source rather than to an accidental interaction.
  • I3. Meaning graphs built close to the surface give paraphrases low structural overlap, so comparing raw graphs falsely signals contradiction; normalization must happen before comparison.
  • I4. A symbolic representation can heighten contrast between minimal pairs, which helps on adversarial sets but makes the system over-reject easy entailments; gains on HANS-like probes should be checked against in-distribution data.
  • I5. Bin accuracy by combined input length, with one catch-all bin for the sparse long tail, to test robustness claims directly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use this as a negative control in evaluation by checking whether the entailment or answer decisions track raw tree or frame overlap more than the truth.
  • I2 (learn): Ablate reader inputs (raw tokens vs UD tree vs frames) under the same setup to see which layer actually carries the answer.
  • I3 (frames): Canonicalize frames before matching premise to hypothesis, for example by aligning passive with active and mapping synonymous predicates to one VerbNet class.
  • I4 (answer): Report test accuracy on both adversarial minimal pairs and ordinary items so that a "say No more often" bias is not mistaken for understanding.
  • I5 (tree): Break down parser LAS and downstream answer accuracy by sentence length to see where gparse or the perceptron parser degrades.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations🎓Amherst CollegeAmherst Center · United States4 works · 13 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSemantic parsing & meaning representation · 185 works

Cite

Cite as Junghyun Min, Xiulin Yang and Shira Wein. 2025. When Does Meaning Backfire? Investigating the Role of AMRs in NLI. Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.starsem-1.16

@inproceedings{min2025when,
  title = {{When Does Meaning Backfire? Investigating the Role of AMRs in NLI}},
  author = {Junghyun Min, Xiulin Yang and Shira Wein},
  booktitle = {Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.starsem-1.16},
  url = {https://doi.org/10.18653/v1/2025.starsem-1.16}
}

References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze. 2020. Learning as abduction: Trainable natural logic theorem prover for natural language inference. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 20–31, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. In Proceedings of AAAI. ↗
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  5. Zeming Chen, Qiyue Gao, and Lawrence S. Moss. 2021. NeuralLog: Natural language inference with joint neural and logical reasoning. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 78–88, Online. Association for Computational Linguistics. find
  6. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069–4082, Hong Kong, China. Association for Computational Linguistics. ↗
  7. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches – erratum. Natural Language Engineering, 16(1):105–105. ↗
  8. Zahra Delbari and Mohammad Taher Pilehvar. 2025. Beyond accuracy: Revisiting out-of-distribution generalization in NLI models. In The SIGNLL Conference on Computational Natural Language Learning. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Ritam Dutt, Shounak Sural, and Carolyn Rose. 2025. Can dependency parses facilitate generalization in language models? a case study of cross-lingual relation extraction. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 338– 358, Albuquerque, New Mexic, USA. Association for Computational Linguistics. find
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  12. Xuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini, and Pontus Stenetorp. 2024. Using natural language explanations to improve robustness of in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13477–13499, Bangkok, Thailand. Association for Computational Linguistics. find
  13. Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown. 2023. Improving long dialogue summarization with semantic graph representation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13851–13883, Toronto, Canada. Association for Computational Linguistics. find
  14. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗
  15. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8018–8025.
  16. Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, and Mona Diab. 2024. Analyzing the role of semantic representations in the era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3781–3798, Mexico City, Mexico. Association for Computational Linguistics. ↗
  17. Pride Kavumba, Ana Brassard, Benjamin Heinzerling, and Kentaro Inui. 2023. Prompting for explanations improves adversarial NLI. is this true? {yes} it is {true} because {it weakens superficial cues}. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2165–2180, Dubrovnik, Croatia. Association for Computational Linguistics.
  18. Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and 1 others. 2021. Abstract meaning representation (amr) annotation release 3.0. find
  19. Wai Ching Leung, Shira Wein, and Nathan Schneider. 2022. Semantic similarity as a window into vector-and graph-based metrics. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 106–115, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  21. R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 217–227, Online. Association for Computational Linguistics. ↗
  22. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  23. Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339–2352, Online. Association for Computational Linguistics.
  24. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy. Association for Computational Linguistics. find
  25. Bill Noble, Rasmus Blanck, and Gijs Wijnholds. 2025. In the mood for inference: Logic-based natural language inference with large language models. In Proceedings of the 5th Workshop on Natural Logic Meets Machine Learning (NALOMA), pages 33–47, Bochum, Germany. Association for Computational Linguistics. find
  26. Juri Opitz. 2023. SMATCH++: Standardized and extended evaluation of semantic graphs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1595–1607, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  27. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  28. Juri Opitz, Shira Wein, Julius Steen, Anette Frank, and Nathan Schneider. 2023. AMR4NLI: Interpretable and robust NLI measures from semantic graphs. In Proceedings of the 15th International Conference on Computational Semantics, pages 275–283, Nancy, France. Association for Computational Linguistics. ↗
  29. Hongji Li, Andrianos Michail, Reto Gubelmann, Simon Clematide, and Juri Opitz. 2025. Sentence smith: Formally controllable text transformation and its application to evaluation of text embedding models. arXiv preprint arXiv:2502.14734. ↗
  30. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, on Empirical Methods in Natural Language Processing, pages 6856–6875, Miami, Florida, USA. Association for Computational Linguistics. pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  31. Zacchary Sadeddine, Juri Opitz, and Fabian Suchanek. 2024. A survey of meaning representations – from theory to practical utility. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2877–2892, Mexico City, Mexico. Association for Computational Linguistics. find
  32. Sofia Serrano, Jesse Dodge, and Noah A. Smith. 2023. Stubborn lexical bias in data and models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8131–8146, Toronto, Canada. Association for Computational Linguistics. find
  33. Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, and Yanjun Qi. 2021. Perturbing inputs for fragile interpretations in deep natural language processing. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 420–434, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  34. Saurabh Srivastava, Sweta Pati, and Ziyu Yao. 2025. Instruction-tuning llms for event extraction with annotation guidelines. arXiv preprint arXiv:2502.16377. ↗
  35. Saurabh Srivastava and Ziyu Yao. 2025. Revisiting prompt optimization with large reasoning models-a case study on event extraction. arXiv preprint arXiv:2504.07357. ↗
  36. Sarah Uhrig, Yoalli Garcia, Juri Opitz, and Anette Frank. 2021. Translate, then parse! a strong baseline for cross-lingual AMR parsing. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 58–64, Online. Association for Computational Linguistics. find
  37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. ↗
  38. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. Association for Computational Linguistics. ↗
  39. Shira Wein and Juri Opitz. 2024. A survey of AMR applications. In Proceedings of the 2024 Conference Shira Wein and Nathan Schneider. 2024. Lost in translationese? reducing translation effect using Abstract Meaning Representation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 753–765, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  40. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  41. Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, T. J. Hazen, and Alessandro Sordoni. 2021. Increasing robustness to spurious correlations using forgettable examples. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3319–3332, Online. Association for Computational Linguistics. find
  42. Hitomi Yanaka, Koji Mineshima, Pascual Martínez-Gómez, and Daisuke Bekki. 2018. Acquisition of phrase correspondences using natural deduction proofs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 756–766, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  43. Peiran Yao, Kostyantyn Guzhva, and Denilson Barbosa. 2024. Semantic graphs for syntactic simplification: A revisit from the age of LLM. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 105–115, Bangkok, Thailand. Association for Computational Linguistics. find
  44. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198. ↗
  45. Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. 2020. The curse of performance instability in analysis datasets: Consequences, source, and suggestions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8215–8228, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2025-when-does-meaning-backfire-investigating/