Elamur Research
☰
home / papers / 2019 / The CommitmentBank: Investigating projection in naturally occurring discourse

The CommitmentBank: Investigating projection in naturally occurring discourse

What it is about

The paper builds a corpus of 1,200 naturally occurring discourses, from the Wall Street Journal, the fiction part of the BNC and Switchboard dialogue. In each one, the last sentence has a clause-embedding predicate (such as "know", "think", "tell") under negation, a modal, a question or a conditional, and the question is whether the speaker is still committed to the content of the complement. The workers on Mechanical Turk (at least eight per item) answered on a scale from -3 (speaker is certain the complement is false) to +3 (certain it is true), and 982 epistemic-modal items with 45 predicates remain for the analysis. The factive/nonfactive split explains some of the variance (a coefficient of 1.60 for factivity, with a Nagelkerke R² of 0.126), but the predicate lemma explains more (R² 0.228). Subject person (0.124), tense (0.078) and genre (0.062) add little alone, and all factors together reach R² 0.35. Context-only plausibility, from 558 items, explains very little (R² 0.036). The main finding is that projection is gradient and item-specific, so the factive class is a weak predictor on its own.

Ideas

For Mova

Chapters: languages/en/33-implicature-and-presupposition.md (projection and presupposition), languages/en/27-negation.md (the negated embedding), languages/en/26-questions.md (complements under a question), and languages/en/31-speech-acts.md (commitment). Code: prag/src/feats.rs, prag/src/lens.rs, the narration frames in world/src/story.rs (a complement becomes a claim of the speaker in a frame, not a fact of the main line), en/src/parse.rs for the complement clauses. Licence note: the paper is CC BY 4.0, so short quotes with the citation are fine. The CommitmentBank data is a separate resource on GitHub, and its licence has to be checked before any use. Until then, treat it as a test set only.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — ojs.ub.uni-konstanz.de/sub/index.php/sub/article/download/601/456

Summary

The paper asks when listeners take a speaker to be committed to the content of a finite clausal complement (CC) that sits under an entailment-canceling operator such as negation, a modal, a question or a conditional antecedent. This phenomenon is called projection. The traditional answer is lexical: factive predicates (e.g., know) let their complements project, and nonfactive ones do not. To test this on real usage, the authors built the CommitmentBank. It contains 1,200 naturally occurring discourse segments drawn automatically from WSJ, BNC fiction and Switchboard, then checked by hand. Crowdworkers on Mechanical Turk rated how certain the speaker is of each CC. Ordinal mixed-effects models show that predicate, embedding environment, genre, tense, person and contextual plausibility all matter. Even so, these factors together explain only a modest share of the variance, and the factive/nonfactive split alone has weak predictive power. The resource matters for two reasons:

  • It gives theorists an empirical testbed for projection.
  • It supports NLP tasks such as information extraction, where telling whether a speaker endorses or denies an embedded claim is crucial.

Key points

  • Data: 1,200 discourses covering 48 clause-embedding predicates. Each target sentence comes with up to two preceding context sentences or turns.
  • Extraction: candidates were pulled from Stanford dependency parses and then manually curated. Think is by far the most frequent predicate.
  • Annotation: each item was judged by at least eight self-reported native English speakers on MTurk, using a −3 to +3 certainty scale. Each questionnaire held eight target items plus two constructed attention-check controls.
  • Plausibility: for 558 WSJ/BNC items, the authors also collected ratings of how plausible the CC is given the context alone. Used on its own as a predictor, plausibility explains very little (Nagelkerke R² = 0.036).
  • Combined model: a model with embedding×genre, tense×person and predicate lemma reaches Nagelkerke R² = 0.35, versus 0.29 when factivity replaces the lemma. Adding plausibility on the 558-item subset gives 0.31 (0.24 with factivity). The corpus is released on GitHub (mcdm/CommitmentBank).

Interesting ideas

  • Bipolar rating scale: the scale lets annotators say the speaker is committed to the falsity of the complement, not just to its truth or to uncertainty. This captures "counterfactive" readings that a one-sided factuality scale would miss, which is useful for any veridicality or stance dataset.
  • Harvesting via syntactic patterns: candidates were mined with dependency-parse queries (predicate with a clausal complement under negation, modal, question or conditional) and then hand-filtered. This is a cheap way to collect naturally occurring examples of a specific construction across genres, applicable to other languages that have parsers.
  • Asking untrained speakers an intuitive question: instead of asking about presupposition, annotators answered a plain question about how sure the speaker is. This turns a theoretical notion into a judgment laypeople can make reliably, a useful design for crowdsourcing semantic phenomena.
  • Context-only control annotation: a separate rating of how plausible the complement is without the target sentence lets researchers separate the predicate's contribution from priors set by the context.
  • Testing a theory's categories against a fine-grained baseline: the authors compared explained variance with factivity versus the individual predicate lemma as the predictor. This shows how much a theory's coarse categories lose against the finer-grained alternative, and the same method could be used to evaluate any linguistic taxonomy.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18148/sub/2019.v23i2.601

@inproceedings{marneffe2019commitmentbank,
  title = {{The CommitmentBank: Investigating projection in naturally occurring discourse}},
  author = {Marneffe, Marie-Catherine de and Simons, Mandy and Tonhauser, Judith},
  booktitle = {Proceedings of Sinn und Bedeutung 23 (2019), vol. 2, pp. 107–124},
  year = {2019},
  doi = {10.18148/sub/2019.v23i2.601},
  url = {https://doi.org/10.18148/sub/2019.v23i2.601}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anand, P. and V. Hacquard (2014). Factivity, belief and discourse. In The Art and Craft of Semantics: A Festschrift for Irene Heim, pp. 69–90. Cambridge, MA: MIT Working Papers in Linguistics. find
  2. Beaver, D. (2010). Have you noticed that your belly button lint colour is related to the colour of your clothing? In R. Bäuerle, U. Reyle, and T. E. Zimmermann (Eds.), Presuppositions and Discourse: Essays Offered to Hans Kamp, pp. 65–99. Leiden, The Netherlands: Brill. find
  3. Chierchia, G. and S. McConnell-Ginet (1990). Meaning and Grammar. Cambridge, MA: MIT Press. find
  4. Christensen, R. H. B. (2018). “ordinal”: Regression models for ordinal data. R package version 2018.8-25. http://www.cran.r-project.org/package=ordinal/. de Marneffe, M.-C., B. MacCartney, and C. D. Manning (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pp. 449–454. ↗ our card
  5. Gazdar, G. (1979). A solution to the projection problem. In C.-K. Oh and D. A. Dinneen (Eds.), Syntax and Semantics, Volume 11: Presupposition, pp. 57–89. New York: Academic Press. find
  6. Heim, I. (1983). On the projection problem for presuppositions. In M. Barlow, D. Flickinger, and M. Westcoat (Eds.), West Coast Conference on Formal Linguistics 2, pp. 114–125. find
  7. Karttunen, L. (1971). Some observations on factivity. Papers in Linguistics 4, 55–69. find
  8. Kiparsky, P. and C. Kiparsky (1970). Fact. In M. Bierwisch and K. E. Heidolph (Eds.), Progress in linguistics, pp. 143–173. The Hague, Paris: Mouton. find
  9. Klein, D. and C. D. Manning (2003). Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association of Computational Linguistics, pp. 423–430. find our card
  10. R Development Core Team (2016). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. find
  11. Schlenker, P. (2010). Local contexts and local meanings. Philosophical Studies 151, 115–142. find
  12. Simons, M. (2007). Observations on embedding verbs, evidentiality, and presupposition. Lingua 117(6), 1034–1056. find
  13. Smith, E. A. S. and K. C. Hall (2014). The relationship between projection and embedding environment. In 48th Annual Meeting of the Chicago Linguistic Society, Volume 48(1), pp. 563–578. find
  14. Stalnaker, R. C. (1974). Pragmatic presuppositions. In M. Munitz and P. Unger (Eds.), Semantics and Philosophy, pp. 197–213. New York: New York University Press. find
  15. Tonhauser, J. (2016). Prosodic cues to presupposition projection. Semantics and Linguistic Theory 26, 934–960. find
  16. Tonhauser, J., D. Beaver, and J. Degen (2018). How projective is projective content? Gradience in projectivity and at-issueness. Journal of Semantics 35, 495–542. van der Sandt, R. (1992). Presupposition projection as anaphora resolution. Journal of Semantics 9, 333–377. find