Elamur Research⌕
☰
home / papers / 2019 / Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment

Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment

Paper · 2019 · Nanjiang Jiang and Marie-Catherine de Marneffe

Summary

The paper asks whether leading speaker-commitment (event factuality) systems can tell how strongly a speaker stands behind an embedded clause when the sentence is natural, varied text rather than tidy newswire. The authors test two systems on the CommitmentBank, a corpus of naturally occurring discourse in which the complement of a clause-embedding verb sits under negation, a modal, a question or a conditional. The first is the rule-based annotator of Stanovsky et al. (2017), which applies implicative signatures over a dependency tree. The second is the biLSTM family of Rudinger et al. (2018): linear, tree and hybrid, trained multi-task on four factuality corpora. Both systems score much worse here than on their usual benchmarks. On the full set, the rule-based system beats the neural ones. A breakdown by linguistic feature shows the failure points:

  • conditionals,
  • neg-raising,
  • the uncommitted middle of the scale.

The authors conclude that robust inference needs more built-in linguistic knowledge and broader coverage of constructions.

Key points

  • The corpus. CommitmentBank has 1,200 items drawn from WSJ news, British National Corpus fiction and Switchboard dialog.
    • Each item consists of a single target sentence accompanied by no more than two sentences of context.
    • At least eight Mechanical Turk workers rated each item on a −3 to +3 scale, and the mean rating is the gold score.
  • The high-agreement subset. Items are kept only if at least 80% of their ratings fall in the same band (committed, uncommitted or committed-to-not).
    • This leaves 556 items covering 37 verbs.
    • Krippendorff's α rises from 0.53 on the full set to 0.74 on the subset.
  • The neural setup. The biLSTMs were trained multi-task on FactBank, UW, MEANTIME and UDS, with shared encoder weights and separate regression heads per corpus. Cross-validated ridge regression then merged the four outputs. A constant baseline that always predicts −2.0 was also included.
  • Results by environment. On the subset:
    • the rule-based model correlates well only under negation (r = 0.45);
    • the hybrid model sits near r ≈ 0.25 for negation, modals and questions;
    • neither model handles conditionals;
    • neg-raising items show essentially no correlation.
  • How the models fail. Rule-based predictions bunch at ±3, so they miss gradience. Hybrid predictions are mostly positive, so they miss negative commitment. When outputs are clustered into three classes, neither model detects the uncommitted class.

Interesting ideas

  • I1. Build a "clean" evaluation subset by grouping graded ratings into bands and keeping items where most annotators agree, then confirm that the subset's feature mix matches the full set, so it stays representative.
  • I2. Report scores separately for each linguistic slice (embedding environment, genre, factive vs. nonfactive verb, neg-raising) instead of one aggregate, which exposes where a system is blind.
  • I3. Compute factuality by walking the dependency tree top-down and flipping polarity according to verb signatures (e.g., refuse flips its complement) and the scope of negation or uncertainty markers.
  • I4. Treat neg-raising (a negated think/believe read as negating the complement) as its own inference pattern, because both systems fail on it.
  • I5. Turn regression outputs into classes by fitting three Gaussian-mixture clusters to gold scores and predictions; this shows whether a model ever produces the uncommitted middle.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use the same filter on silver data from the witness parsers, keeping high-agreement items as the trusted test core and checking that construction distributions are preserved.
  • I2 (learn): Tag evaluation items by construction and track per-slice correlation and error, so regressions on rare constructions such as conditionals become visible.
  • I3 (frames): A transparent rule pass over UD trees could assign a commitment value to each embedded frame, with every flip recorded as an explanation step.
  • I4 (frames): Add an explicit neg-raising rule for verbs like think and believe in prag, so "I don't believe p" yields a negative commitment to p in the frames.
  • I5 (learn): Use this as a diagnostic so graded certainty scores in the text world are checked for a real "unknown" state, not just true/false.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 685 works

Cite

Cite as Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1412

@inproceedings{jiang2019do,
  title = {{Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment}},
  author = {Nanjiang Jiang and Marie-Catherine {de Marneffe}},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1412},
  url = {https://doi.org/10.18653/v1/p19-1412}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Beaver. 2010. Have you noticed that your belly button lint colour is related to the colour of your clothing? In Rainer Bäuerle, Uwe Reyle, and Thomas Ede Zimmermann, editors, Presuppositions and Discourse: Essays Offered to Hans Kamp, pages 65–99. Leiden, The Netherlands: Brill. find
  2. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. Linguistic Data Consortium, Philadelphia, PA. find
  3. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop, pages 68–73. find
  4. Irene Heim. 1983. On the projection problem for presuppositions. In West Coast Conference on Formal Linguistics (WCCFL) 2, pages 114–125. find
  5. Lauri Karttunen. 1971. Some observations on factivity. Papers in Linguistics, 4:55–69. ↗
  6. Lauri Karttunen. 1973. Presuppositions and compound sentences. Linguistic Inquiry, 4(2):169–193. find
  7. Lauri Karttunen. 2012. Simple and phrasal implicatives. In Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 124–131. find
  8. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648. ↗
  9. Mandy Simons, Judith Tonhauser, David Beaver, and Craige Roberts. 2010. What projects and why. In Proceedings of Semantics and Linguistic Theory 20. CLC Publications. ↗
  10. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357.
  11. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724. ↗
  12. Amnon Lotan, Asher Stern, and Ido Dagan. 2013. TruthTeller: Annotating predicate truth. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 752–757. find
  13. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23. ↗
  14. Anne-Lyse Myriam Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the newsreader multilingual event and time corpus. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 4417–4422. ↗
  15. Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg, Owen Rambow, Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gregory Werner, Yorick Wilks, and Janyce Wiebe. 2015. A new dataset and evaluation for belief/factuality. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 82–91. find
  16. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pages 731–744.
  17. Roser Saurı́ and James Pustejovsky. 2009. FactBank: A corpus annotated with event factuality. Language resources and evaluation, 43(3):227. find
  18. Roser Saurı́ and James Pustejovsky. 2012. Are you sure that this happened? Assessing the factuality degree of events in text. Computational Linguistics, 38(2):261–299.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2019-do-you-know-that-florence/