Elamur Research⌕
☰
home / papers / 2022 / Context Matters: A Pragmatic Study of PLMs' Negation Understanding

Context Matters: A Pragmatic Study of PLMs' Negation Understanding

Summary

Earlier NLP studies concluded that pretrained transformers largely ignore negation, based on context-free minimal pairs such as "Birds can/cannot fly." Drawing on the pragmatic tradition in linguistics, Gubelmann and Handschuh argue that such probes confuse two failures: not grasping negation, and not inferring a default context in which a negated sentence would make sense. To separate them, they build masked language modelling tasks in which a short micro-context precedes the masked slot. The test is whether a model avoids predicting a verb that the preceding sentence negated. The templates are designed so that known shallow cues do not help, and the data are tailored to each tested model. Factors such as gender, first name, profession, syntax, model-specific probability rank, misleading primes and shifts in reference are varied systematically. Fine-tuned variants of the strongest models are also evaluated. Almost all models proved sensitive to negation, and fine-tuned RoBERTa-large performed robustly. The authors therefore paint a considerably more optimistic picture of PLMs' negation understanding than earlier semantics-based work. They also stress that context must be controlled when probing models.

Key points

  • The resource consists of 43 hand-crafted templates, each checked together with Dr. David Dolby, who is both a native speaker and a language philosopher. Expanded per model, they yield roughly 12 million test sentences.
  • Tested families include BERT (base/large cased), RoBERTa, XLNet and DistilBERT, plus fine-tuned "neg-" variants of the best models.
  • For neg-roberta-large, the tendency to repeat an activity verb fell by about 44% when that verb was negated in the context.
  • Under misleading primes, neg-roberta-large's error rate stayed at or below 12%, rising to 30% only when a random sentence was inserted. Over the same insertions, neg-xlnet-large's error rate climbed from 8% to 43% and beyond.
  • The context-specific probability rank of a token strongly affects errors, especially for BERT, with most of the drop occurring between ranks 0 and 50. When the subject clearly changes, the RoBERTa models and fine-tuned XLNet repeat the verb with more than 85% probability, showing that they track reference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.315

@inproceedings{gubelmann2022context,
  title = {{Context Matters: A Pragmatic Study of PLMs' Negation Understanding}},
  author = {Gubelmann, Reto and Handschuh, Siegfried},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.315},
  url = {https://doi.org/10.18653/v1/2022.acl-long.315}
}

References

44 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. Proceedings ofthe 58th Annual Meeting ofthe Association for Computational Linguistics, pages 5185–5198. ↗ our card
  2. Wayne Davis. 2019. Implicature. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Fall 2019 edition. Metaphysics Research Lab, Stanford University. find
  3. Wayne A Davis. 2016. Irregular negatives, implicatures, and idioms. Springer. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. ↗ our card
  6. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899. ↗
  7. Aditya Khandelwal and Suraj Sawant. 2019. Negbert: A transfer learning approach for negation detection and scope resolution. arXiv preprint arXiv:1911.04211. ↗
  8. Gottlob Frege. 1892. Über sinn und bedeutung. Zeitschrift für Philosophie und philosophische Kritik, 100:25–50. find
  9. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–4374, Hong Kong, China. Association for Computational Linguistics. find our card
  10. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. ↗
  11. Hans-Johann Glock. 2019. Agency, intelligence and reasons in animals. Philosophy, pages 1–27. ↗
  12. H. P. Grice and P. F. Strawson. 1956. In defense of a dogma. 65(2):141–150. find
  13. H Paul Grice. 1978. Further notes on logic and conversation. In Pragmatics, pages 113–127. Brill. ↗
  14. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. find
  16. Laurence R. Horn. 2001. A natural history of negation. CSLI. find
  17. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106–9118. find our card
  18. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  19. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics. find our card
  20. Barbara Kaup. 2009. How are pragmatic differences between positive and negative sentences captured in the processes and representations in language comprehension. In Semantics and Pragmatics: From Experiment to Theory. Palgrave Macmillan. find
  21. Peter Lipton. 2004. Inference to the Best Explanation, 2 edition. Routledge. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  23. David Marr. 2010 [1982]. Vision. The MIT Press. find
  24. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. find our card
  25. Richard Moot and Christian Retoré. 2019. Natural language semantics and computability. Journal of Logic, Language and Information, 28(2):287–307. ↗
  26. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664. find
  27. Ira A Noveck. 2009. Meaning and inference linked to negation: An experimental pragmatic approach. In Semantics and Pragmatics: From Experiment to Theory. Palgrave MacMillan. find
  28. Gerhard Preyer. 2018. Beyond Semantics and Pragmatics. Oxford University Press. find
  29. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguistics. find
  30. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. ↗ our card
  31. Bertrand Russell. 1905. 14(56):479–493. On denoting. find
  32. Mind, Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  33. John Searle. 1980. Minds, brains, and programs. Behavioral and brain sciences, 3(3):417–424. ↗
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  35. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, Italy. Association for Computational Linguistics. find our card
  36. Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. ↗ our card
  37. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763. ↗ our card
  38. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. ↗
  39. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162. find
  40. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems. find our card
  41. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  42. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. ↗ our card
  43. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv:1909.10430. ↗
  44. Ludwig Wittgenstein. 2006/1953. Philosophische untersuchungen. In Werkausgabe Band 1. Suhrkamp. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gubelmann-2022-context-matters-pragmatic-study-plms/