Elamur Research⌕
☰
home / papers / 2023 / PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments

PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments

Paper · 2023 · Daiki Asami and Saku Sugawara

Summary

Presuppositions are background assumptions a speaker treats as already given. They differ from ordinary entailments because they tend to survive when a sentence is negated, questioned, put in a conditional or modalized. Linguists have shown that this "projectivity" is not uniform, but earlier NLI probes either had no human baseline (IMPPRES) or tested only negation (NOPE). The authors first collect crowd judgments on a 900-pair subset of IMPPRES. They then build PROPRES, a template-generated NLI dataset of 12,000 premise–hypothesis pairs. It crosses six new trigger types, chosen with some lexical variety, with five environments. Each pair is tested against both a target hypothesis (the presupposition) and a control hypothesis (the entailment). Dense human labels on a 600-pair subset show that projectivity is graded and depends on how the trigger and environment combine, including cases not previously described. Four models were tested: bag-of-words, InferSent, RoBERTa and DeBERTa. Even the strongest, DeBERTa, does poorly on controls and misses most of the human variation. The work argues that pragmatic benchmarks need human-variability baselines and systematic crossing of linguistic factors.

Key points

  • PROPRES design: 12k pairs built from six triggers crossed with five environments.
    • Triggers: iterative again, aspectual verbs, manner adverbs, factives, comparatives, temporal adverbs.
    • Environments: unembedded, negation, interrogative, conditional, modal.
    • Data and generation code are released at github.com/nii-cl/projectivity-of-presupposition.
  • Human labels:
    • IMPPRES subset: 900 pairs, about 9.4 labels each.
    • PROPRES subset: 600 pairs, more than 50 labels each.
    • Collection used MTurk and PCIbex.
  • Graded projectivity in humans: rates range from 55.1% to 99.8%. Graded behaviour appears in 6 of 24 new conditions, against 2 of 24 for IMPPRES.
    • Manner adverbs project weakly in all four embedding environments: 58.3% (negation), 66.6% (interrogative), 62.0% (conditional), 55.1% (modal).
    • Within manner adverbs, rates also vary by word, from 43.3% for angrily to 66.6% for easily.
    • Temporal adverbs and comparatives project weakly only under modals (54.7% and 57.4%).
  • DeBERTa vs. humans:
    • DeBERTa scores just 14.5% on unembedded comparatives, so that trigger was excluded from its later analysis.
    • For again under negation, DeBERTa treats the presupposition as projecting 95% of the time, while humans do so at only 75.8%.
  • IMPPRES unembedded baseline: humans themselves are fairly inaccurate on some IMPPRES triggers, including change-of-state (66.3%), cleft uniqueness and possessive uniqueness. This points to flawed items rather than model failure.

Interesting ideas

  • I1. Each premise is paired with a target hypothesis (the presupposition) and a control hypothesis (the plain entailment), whose labels must diverge once embedded; a system that gets the target right but the control wrong is exposed as pattern-matching.
  • I2. Before reading anything into embedded results, the authors check that humans and models handle the unembedded trigger; triggers that fail this sanity step are dropped rather than misinterpreted as projectivity failures.
  • I3. A full grid of trigger × environment exposed variability that single-environment datasets could not reveal, showing that projection is a property of the combination, not of the trigger alone.
  • I4. The weak projection of again under negation is traced to a scope ambiguity: again can apply either above or below the negation, yielding two opposite presuppositions.
  • I5. Since text lacks prosodic focus, manner-adverb presuppositions become uncertain, so many human labels are used to estimate a graded strength rather than a single gold label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build presupposition tests as target/control pairs, so that a correct answer only counts as success when the matched negative control is also handled correctly.
  • I2 (learn): Gate evaluation per trigger, so that recognition of a bare presupposition must pass before scoring how it survives negation, questions or if-clauses.
  • I3 (frames): In crate prag, store projection behaviour as a table indexed by trigger class and embedding operator (read off the UD tree), each with a confidence, instead of one global "presuppositions project" rule.
  • I4 (world): When building the world of the text, produce both scope readings of again + not as alternative background states and let later context choose, rather than committing to one fact.
  • I5 (answer): Let readers mark facts from manner-adverb presuppositions under negation, questions or modals as defeasible, answering "probably" instead of asserting them as certain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of DelawareNewark · United States20 works · 1,056 citations🔬National Institute of InformaticsTokyo · Japan38 works · 3,247 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Implicature · 138 works

Cite

Cite as Daiki Asami and Saku Sugawara. 2023. PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments. Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.conll-1.9 · arXiv 2312.08755

@inproceedings{asami2023propres,
  title = {{PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments}},
  author = {Daiki Asami and Saku Sugawara},
  booktitle = {Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)},
  year = {2023},
  doi = {10.18653/v1/2023.conll-1.9},
  eprint = {2312.08755},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.conll-1.9}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Clinton Bale. 2007. Quantifiers and verb phrases: An exploration of propositional complexity. Natural Language & Linguistic Theory, 25:447–483. ↗
  2. DeBERTa’s performance does not mirror hu-David I. Beaver. 1997. Presupposition. In Johan van mans’ in some cases. It predicts that the manner Benthem and Alice ter Meulen, editors, Handbook of adverbs in the negation and conditional (P 1 and P 3 logic and language, pages 939–1008. MIT Press and in (7), respectively) are not projective (8.5% and North-Holland, Cambridge, MA and Amsterdam. 14%, respectively), contrary to humans (58.3% and 62.0%, respectively). This indicates that either De- Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. 2018. Let’s do it “again”: A first BERTa lacks the knowledge of these two cases or computational approach to detecting adverbial preprocesses them as if the subject is focalized (e.g., supposition triggers. In Proceedings of the 56th An-did the MAN hurt others seriously?). nual Meeting of the Association for Computational 130 Linguistics (Volume 1: Long Papers), pages 2747– 2755, Melbourne, Australia. Association for Computational Linguistics. find
  3. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  4. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23, volume 2, pages 107–124. ↗
  5. Judith Degen and Judith Tonhauser. 2021a. Are there factive predicates? an empirical investigation. Ling-Buzz. ↗
  6. Judith Degen and Judith Tonhauser. 2021b. Prior beliefs modulate projection. Open Mind, 5:59–70. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Edward Gibson and Evelina Fedorenko. 2013. The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 28(1-2):88–124. find
  9. Jeroen Groenendijk and Martin Stokhof. 1984. Studies on the semantics of questions and the pragmatics of answers. Ph.D. thesis, University of Amsterdam dissertation. find
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654. ↗
  12. Irene Heim. 1983. On the conversational basis of some presuppositions. In Proceedings of the 2nd West Coast Conference on Formal Linguistics, pages 114– 125, Stanford, CA. Stanford Linguistics Association. find
  13. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗
  14. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics, 10:1357–1374. ↗
  15. Lauri Karttunen. 1971. Some observations on factivity. Papers in Linguistics, 4:55–69. ↗
  16. Lauri Karttunen. 1973. Presuppositions of compound sentences. Linguistic inquiry, 4(2):169–193. find
  17. Stephen C. Levinson. 1983. Pragmatics. Cambridge University Press, Cambridge, UK. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  19. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗
  20. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
  21. Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 349–366, Online. Association for Computational Linguistics. ↗
  22. Ellie Pavlick. 2022. Semantic structure in deep learning. Annual Review of Linguistics, 8. ↗
  23. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  25. Christopher Potts. 2015. Presupposition and implicature. In Shalom Lappin and Chris Fox, editors, The handbook of contemporary semantic theory, volume 2, pages 168–202. Wiley-Blackwell, Oxford, UK. ↗
  26. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and pragmatics, 5:6–1. ↗
  27. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics. ↗
  28. Karin Sevegnani, David M. Howcroft, Ioannis Konstas, and Verena Rieser. 2021. OTTers: One-turn topic transitions for open-domain dialogue. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2492–2504, Online. Association for Computational Linguistics. find
  29. Mandy Simons. 2001. On the conversational basis of some presuppositions. In Proceedings of Semantics and Linguistics Theory XI, pages 431–448, Ithaca, NY. CLC Publications. ↗
  30. Robert Stalnaker. 1974. Pragmatic presuppositions. In Milton K. Munitz and Peter K. Unger, editors, Semantics and Philosophy, pages 135–148. New York University Press, New York. find
  31. Jon Stevens, Marie-Catherine de Marneffe, Shari R Speer, and Judith Tonhauser. 2017. Rational use of prosody predicts projection in manner adverb utterances. In 39th Annual Meeting of the Cognitive Science Society, pages 1144–1149. find
  32. Symon Stevens-Guille, Aleksandre Maskharashvili, Amy Isard, Xintong Li, and Michael White. 2020. Neural NLG for methodius: From RST meaning representations to texts. In Proceedings of the 13th International Conference on Natural Language Generation, pages 306–315, Dublin, Ireland. Association for Computational Linguistics. find
  33. Judith Tonhauser, David I. Beaver, and Judith Degen. 2018. How projective is projective content? gradience in projectivity and at-issueness. Journal of Semantics, 35(3):495–542. editors, The Lexicon in the Organization of Language, pages 81–118. John Benjamins, Amsterdam/Philedelphia. ↗
  34. Aaron S White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, pages 221–234. find
  35. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  37. Hitomi Yanaka and Koji Mineshima. 2021. Assessing the generalization capacity of pre-trained language models through Japanese adversarial natural language inference. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 337–349, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  38. Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagreement in natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908–4915, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asami-2023-propres-investigating-projectivity-presupposition-various/