Elamur Research⌕
☰
home / papers / 2023 / Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions

Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions

Paper · 2023 · Jad Kabbara and Jackie Cheung

Summary

NLI models fine-tuned on MNLI tend to fail on inferences driven by presupposition, such as the fact that "X won the match" takes for granted that X played. Linguistic theory ties presupposition closely to discourse structure and to sarcasm. The authors test whether that link can be used for transfer learning. Before the usual MNLI fine-tuning, they give BERT-style models an extra "pre-finetuning" stage on one of three tasks (classifying the relations between discourse units, modeling how coherent a text is, or detecting sarcasm), using two datasets for each task. They then evaluate on the presupposition portion of ImpPres. This extra stage generally raised accuracy on presupposition cases. To check whether the gain came simply from more training data, they ran control pre-finetuning on POS tagging and NER, which mostly hurt performance. The main claim is that the kind of task matters, not only the extra data. This suggests that pragmatics-oriented intermediate training is a practical way to make inference models more sensitive to what is taken for granted.

Key points

  • Pre-finetuning covers three tasks: identifying discourse relations, judging text coherence and spotting sarcasm. The datasets named in the analysis are PDTB, a Wiki coherence set and a Reviews sarcasm set.
  • Evaluation uses the presupposition part of ImpPres, a semi-automatically generated resource of 25.5k items in SNLI/MNLI format. It has 9 presupposition sub-datasets of 1900 samples each, and the models are trained on MNLI (433k pairs).
  • Control tasks were POS tagging on WSJ-PTB and NER on CoNLL-2003. They lowered the average score and the scores on most ImpPres sub-datasets, which supports the view that task choice matters.
  • An error analysis on the Possessed Definites Existence sub-dataset looked at items that pre-finetuning turned from wrong to right (fewer than 60 per setting). Over 95% of these had the gold label neutral. The vanilla model had mostly called them entailment, and sometimes contradiction when the premise contained a negation.
  • Stated limitations:
    • The study covers English only.
    • ImpPres is built from templates over a vocabulary of about 3000 lexical items, so some sentences are unnatural or implausible.

Interesting ideas

  • I1. Pick intermediate training tasks for a theoretical reason (here, presupposition's ties to discourse and sarcasm) rather than piling on many tasks at random; targeted transfer can help where generic extra training does not.
  • I2. Separate "more data" from "the right data" by training equally on unrelated tasks (POS, NER) as a negative control; if the gain disappears, the effect is task-specific.
  • I3. Analyze only the items that flip from wrong to right after an intervention and look at their label distribution; here this exposed a bias toward entailment and a "negation means contradiction" shortcut.
  • I4. Treat presupposition triggers such as possessed definites ("Kenneth's politics") as sources of background facts whose truth is independent of whether the main clause is negated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Training the prag implicature and irony components alongside presupposition handling could let one shared signal about "what is taken for granted" improve all three.
  • I2 (learn): Any evaluation of new silver data or teacher tiers could include a matched-size unrelated-data control, so improvements are not wrongly credited to content.
  • I3 (answer): Comparing flipped answers between reader versions could reveal shortcut cues like negation that the reasoning step is misusing.
  • I4 (world): When building the text world, assert existence facts from possessives and definites even under negation, keeping them apart from what the sentence asserts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksDiscourse analysis · 466 worksImplicature · 138 works

Cite

Cite as Jad Kabbara and Jackie Cheung. 2023. Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.703

@inproceedings{kabbara2023investigating,
  title = {{Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions}},
  author = {Jad Kabbara and Jackie Cheung},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.703},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.703}
}

References

53 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799–5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  2. John Langshaw Austin. 1962. How to do things with words. Clarendon Press. find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  5. Elisabeth Camp. 2012. Sarcasm, pretense, and the semantics/pragmatics distinction. Noûs, 46(4):587– 634. ↗
  6. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. ↗
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, page 177–190, Berlin, Heidelberg. Springer-Verlag. find
  8. Paul C Davis. 2000. Presupposition resolution with discourse information structures. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Filippo Domaneschi. 2016. Introduction: Presuppositions philosophy, linguistics and psychology. Topoi, 35(1):5–8. ↗
  11. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 392–398, Istanbul, Turkey. European Language Resources Association (ELRA). ↗
  12. Gottlob Frege. 1892. Über Sinn und Bedeutung. Zeitschrift für Philosophie und philosophische Kritik, 100:25–50. find
  13. Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 59–69, Dublin, Ireland. Association for Computational Linguistics. find
  14. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  15. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  16. Irene Heim. 1992. Presupposition projection and the semantics of attitude verbs. Journal of semantics, 9(3):183–221. ↗
  17. Sungho Jeon and Michael Strube. 2022. Entity-based neural local coherence modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7787–7805, Dublin, Ireland. Association for Computational Linguistics. ↗
  18. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗
  19. Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In Proceedings of the 29th International Conference on Computational Linguistics, pages 779–785, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  20. Hans Kamp and U Reyle. 1993. From discourse to logic: introduction to modeltheoretic semantics of natural language, formal logic and discourse representation theory. Part 1. Springer Science & Business Media. find
  21. Robert T. Kasper, Paul C. Davis, and Craige Roberts. 1999. An integrated approach to reference and presupposition resolution. In The Relation of Discourse/Dialogue Structure and Reference.
  22. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceeding of the 2015 International Conference on Learning Representation (ICLR 2015), San Diego, California. ↗
  23. Philippe Laban, Luke Dai, Lucas Bandarkar, and Marti A. Hearst. 2021. Can transformer models measure coherence in text: Re-thinking the shuffle test. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1058–1064, Online. Association for Computational Linguistics. ↗
  24. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 100–109, Taipei, Taiwan. Asian Federation of Natural Language Processing. ↗
  25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  27. Bill MacCartney and Christopher D. Manning. 2008. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), pages 521–528, Manchester, UK. COLING 2008 Organizing Committee. ↗
  28. Christopher Manning. 2006. Local textual inference : It’s hard to circumscribe , but you know it when you see it - and NLP needs it. find
  29. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  30. Rishabh Misra and Prahal Arora. 2019. Sarcasm detection using hybrid neural network. arXiv preprint arXiv:1908.07414. ↗
  31. Joe O’Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 851–864, Online. Association for Computational Linguistics. ↗
  32. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  33. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  34. Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585–10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. pages 2961–2968, Marrakech, Morocco. European Language Resources Association (ELRA).
  35. Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse TreeBank Version 3.0. LDC2019T05. ↗
  36. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online. Association for Computational Linguistics.
  37. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. ↗
  38. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd Schema Challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740. find
  39. Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. “You are grounded!”: Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850–6861, Online. Association for Computational Linguistics. find
  40. Robert Stalnaker. 1973. Presuppositions. Journal of philosophical logic, 2(4):447–457. find
  41. Peter F. Strawson. 1950. 59(235):320–344. On referring. find
  42. Mind, Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5941–5946, Hong Kong, China. Association for Computational Linguistics. ↗
  43. Christopher Potts. 2015. Presupposition and implicature. The handbook of contemporary semantic theory, 2:168–202. ↗
  44. Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1320–1330, Vancouver, Canada. Association for Computational Linguistics. find
  45. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2023. Fine-tuning large neural language models for biomedical natural language processing. Patterns, 4(4).
  46. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. ↗
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc.
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  50. Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019. A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678–687, Florence, Italy. Association for Computational Linguistics. ↗
  51. Deniz Zeyrek, Amália Mendes, Yulia Grishina, Murathan Kurfalı, Samuel Gibbon, and Maciej Ogrodniczuk. 2019. Ted multilingual discourse bank (ted-mdb): A parallel corpus annotated in the PDTB style. Language Resources and Evaluation, pages 1–27. ↗
  52. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199. ↗
  53. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kabbara-2023-investigating-effect-pre-finetuning-bert/