Elamur Research
☰
home / papers / 2012 / Did It Happen? The Pragmatic Complexity of Veridicality Assessment

Did It Happen? The Pragmatic Complexity of Veridicality Assessment

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J12-2003.pdf

Summary

Event and relation extraction systems usually treat any relation they extract as true. They largely ignore veridicality, meaning whether a reader takes a described event to have actually happened. Earlier work mostly treated veridicality as a fixed lexical property, such as believe being non-veridical or realize being veridical. The authors argue that context, source trustworthiness and world knowledge strongly shape these judgments.

To test this, they took sentences from the FactBank corpus, whose labels deliberately exclude world knowledge. Linguistically naive annotators relabeled them under loose instructions to follow their natural reading, with ten people judging each sentence. The new labels often depart from the semantics-only ones, yet they remain systematic. They also support keeping FactBank's seven-way scheme rather than a binary certain/uncertain split. Because annotators frequently disagree, the authors model veridicality as a probability distribution over categories. They then train a classifier that learns and predicts such distributions using lexical cues, structural features and rough proxies for world knowledge. This matters for information extraction, textual entailment and BioNLP, where unreliable or hedged claims should not be extracted as facts.

Key points

  • FactBank (Saurí and Pustejovsky 2009) contains 208 newswire and broadcast-news documents with 9,472 manually identified single-word events (verbs, nouns, adjectives). Its sources are TimeBank 1.2 and part of AQUAINT TimeML.
  • FactBank tags combine a certainty level with a polarity:
    • certainty levels are CT, PR, PS and U;
    • polarity values are +, − and u;
    • each tag is assigned per event–source pair, for example the author or a quoted participant.
  • In the new layer, ten annotators labeled each sentence, which produces label distributions rather than single gold tags.
  • Pragmatic reading often strengthens lexically uncertain cases. Examples include treating "X said S" from a credible source as signalling S, or shifting PS to PR or CT. Even know can occasionally be used non-factively.
  • Text provenance could not be studied because all sentences come from mainstream news outlets such as the WSJ, AP and NYT. The authors leave this for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

70 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00097

@article{marneffe2012did,
  title = {{Did It Happen? The Pragmatic Complexity of Veridicality Assessment}},
  author = {Marneffe, Marie-Catherine de and Manning, Christopher D. and Potts, Christopher},
  journal = {Computational Linguistics},
  year = {2012},
  doi = {10.1162/coli\_a\_00097},
  url = {https://doi.org/10.1162/coli_a_00097}
}

References

57 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asher, Nicholas. 2000. Truth conditional discourse semantics for parentheticals. Journal of Semantics, 17(1):31–50. find
  2. Asher, Nicholas and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press, Cambridge. find
  3. Barwise, Jon. 1981. Scenes and other situations. The Journal of Philosophy, 78(7):369–397. find
  4. Beaver, David. 2010. Have you noticed that your belly button lint colour is related to the colour of your clothing? In Rainer Bäuerle, Uwe Reyle, and Thomas Ede Zimmermann, editors, Presuppositions and Discourse: Essays Offered to Hans Kamp. Elsevier, Philadelphia, PA, pages 65–99. find
  5. Berger, Adam L., Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71. find our card
  6. Chapman, Siobhan. 2005. Paul Grice: Philosopher and Linguist. Palgrave Macmillan, Houndmills, Basingstoke, Hampshire. find
  7. Chapman, Wendy W., Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001. A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5):301–310. find
  8. Clifton, Charles Jr. and Lyn Frazier. 2010. Imperfect ellipsis: Antecedents beyond syntax? Syntax, 13(4):279–297. find
  9. Crouch, Richard, Lauri Karttunen, and Annie Zaenen. 2006. Circumscribing is not excluding: A reply to Manning. Ms., Palo Alto Research Center, Palo Alto, CA. find
  10. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In J. Quinonero-Candela, I. Dagan, B. Magnini, and F. d’Alché-Buc, editors, Machine Learning Challenges, Lecture Notes in Computer Science, volume 3944. Springer-Verlag, New York, pages 177–190. find
  11. de Marneffe, Marie-Catherine, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation, Genoa, Italy, pages 449–454. find our card
  12. Diab, Mona, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop, Singapore, pages 68–73. find
  13. Elkin, Peter L., Steven H. Brown, Brent A. Bauer, Casey S. Husser, William Carruth, Larry R. Bergstrom, and Dietlind L. Wahner-Roedler. 2005. A controlled trial of automated classification of negation from clinical notes. BMC Medical Informatics and Decision Making, 5(13). find
  14. Farkas, Richárd, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning: Shared Task, Uppsala, Sweden, pages 1–12. find our card
  15. Finlay, Stephen. 2009. Oughts and ends. Philosophical Studies, 143(3):315–340. find
  16. Fleiss, Joseph I. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  17. Frazier, Lyn and Charles Clifton, Jr. 2005. The syntax-discourse divide: Processing ellipsis. Syntax, 8(1):121–174. find
  18. Giannakidou, Anastasia. 1994. The semantic licensing of NPIs and the Modern Greek subjunctive. In Language and Cognition 4, Yearbook of the Research Group for Theoretical and Experimental Linguistics. University of Groningen, The Netherlands, pages 55–68. find
  19. Giannakidou, Anastasia. 1995. Weak and strong licensing: Evidence from Greek. In Artemis Alexiadou, Geoffrey Horrocks, and Melita Stavrou, editors, Studies in Greek Syntax. Kluwer, Dordrecht, pages 113–133. find
  20. Giannakidou, Anastasia. 1999. Affective dependencies. Linguistics and Philosophy, 22(4):367–421. find
  21. Giannakidou, Anastasia. 2001. The meaning of free choice. Linguistics and Philosophy, 24(6):659–735. find
  22. Hazlett, Allan. 2010. The myth of factive verbs. Philosophy and Phenomenological Research, 80(3):497–522. find
  23. Hobby, Jonathan L., Brian D. M. Tom, C. Todd, Philip W. P. Bearcroft, and Adrian K. Dixon. 2000. Communication of doubt and certainty in radiological reports. The British Journal of Radiology, 73(873):999–1001. find
  24. Huang, Yang and Henry J. Lowe. 2007. A novel hybrid approach to automated negation detection in clinical radiology reports. Journal of the American Medical Informatics Association, 14(3):304–311. find
  25. Karttunen, Lauri. 1973. Presuppositions and compound sentences. Linguistic Inquiry, 4(2):169–193. find
  26. Karttunen, Lauri and Annie Zaenen. 2005. Veridicity. In Graham Katz, James Pustejovsky, and Frank Schilder, editors, Annotating, Extracting and Reasoning about Time and Events, number 05151 in Dagstuhl Seminar Proceedings, Dagstuhl, Germany. Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany. find
  27. Kim, Jin-Dong, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 shared task on event extraction. In Proceedings of the Workshop on BioNLP: Shared Task, Boulder, Colorado, pages 1–9. find
  28. Kiparsky, Paul and Carol Kiparsky. 1970. Facts. In M. Bierwisch and K. E. Heidolph, editors, Progress in Linguistics. Mouton. The Hague, Paris, pages 143–173. find
  29. Klein, Dan and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association of Computational Linguistics, Sapporo, Japan, pages 423–430. find our card
  30. Levinson, Stephen C. 1995. Three levels of meaning: Essays in honor of Sir John Lyons. In Frank R. Palmer, editor, Grammar and Meaning. Cambridge University Press, Cambridge, pages 90–115. find
  31. Levinson, Stephen C. 2000. Presumptive Meanings: The Theory of Generalized Conversational Implicature. MIT Press, Cambridge, MA. find
  32. Manning, Christopher D. 2006. Local textual inference: it’s hard to circumscribe, but you know it when you see it—and NLP needs it. Ms., Stanford University. find
  33. Manning, Christopher D. and Dan Klein. 2003. Optimization, maxent models, and conditional estimation without magic. In Tutorial at HLT-NAACL 2003 and ACL 2003. Available at http://nlp.stanford.edu/software/classifier.shtml. ↗
  34. Miller, Mark Crispin. 2001. The Bush Dyslexicon. W. W. Norton and Company, New York, NY. find
  35. Montague, Richard. 1969. On the nature of certain philosophical entities. The Monist, volume 2, pages 159–194. find
  36. Morante, Roser and Walter Daelemans. 2009. A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, Boulder, Colorado, pages 21–29. find
  37. Morante, Roser and Caroline Sporleder, editors. 2010. Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, Uppsala, Sweden. find
  38. Prabhakaran, Vinodkumar, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Proceedings of COLING 2010: Poster Volume, Beijing, China, pages 1014–1022. find
  39. Pustejovsky, James, Marc Verhagen, Roser Saurı́, Jessica Littman, Robert Gaizauskas, Graham Katz, Inderjeet Mani, Robert Knippen, and Andrea Setzer. 2006. Timebank 1.2. Linguistic Data Consortium, Philadelphia, PA. find
  40. Pyysalo, Sampo, Filip Ginter, Juho Heimonen, Jari Björne, Jorma Boberg, Jouni Järvinen, and Tapio Salakoski. 2007. BioInfer: A corpus for information extraction in the biomedical domain. BMC Bioinformatics, 8(50). doi:10.1186/ 1471-2105-8-50. find
  41. Rieh, Soo Young. 2010. Credibility and cognitive authority of information. In M. Bates and M. N. Maack, editors, Encyclopedia of Library and Information Sciences, 3rd ed., Taylor and Francis Group, LLC, New York, pages 1337–1344. find
  42. Rooryck, Johan. 2001. Evidentiality, Part I. Glot International, 5(4):3–11. find
  43. Ross, John Robert. 1973. Slifting. In Maurice Gross, Morris Halle, and Marcel-Paul Schützenberger, editors, The Formal Analysis of Natural Languages. Mouton de Gruyter, The Hague, pages 133–169. find
  44. Rubin, Victoria L. 2007. Stating with certainty or stating with doubt: Intercoder reliability results for manual annotation of epistemically modalized statements. In Proceedings of the NAACL-HLT 2007, Rochester, NY, pages 141–144. find
  45. Rubin, Victoria L., Elizabeth D. Liddy, and Noriko Kando. 2005. Certainty identification in texts: Categorization model and manual tagging results. In J. G. Shanahan, Y. Qu, and J. Wiebe, editors, Computing Attitude and Affect in Text: Theory and Applications (The Information Retrieval Series), Springer-Verlag, New York, pages 61–76. find
  46. Sæbo, Kjell Johan. 2001. Necessary conditions in a natural language. In Caroline Féry and Wolfgang Sternefeld, editors, Audiatur Vox Sapientia. A Festschrift for Arnim von Stechow. Akademie Verlag, Berlin, pages 427–449. find
  47. Saurı́, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. thesis, Computer Science Department, Brandeis University. find
  48. Saurı́, Roser and James Pustejovsky. 2009. FactBank: A corpus annotated with event factuality. Language Resources and Evaluation, 43(3):227–268. find
  49. Searle, John R. 1978. Literal meaning. Erkenntnis, 13:207–224. find
  50. Simons, Mandy. 2007. Observations on embedding verbs, evidentiality, and presupposition. Lingua, 117(6):1034–1056. find
  51. Snow, Rion, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, pages 254–263. find
  52. Szarvas, György, Veronika Vincze, Richárd Farkas, and János Csirik. 2008. The BioScope corpus: annotation for negation, uncertainty and their scope in biomedical texts. In BioNLP 2008: Current Trends in Biomedical Natural Language Processing, Columbus, OH, pages 38–45. find
  53. Travis, Charles. 1996. Meaning’s role in truth. Mind, 105(419):451–466. find
  54. von Fintel, Kai and Sabine Iatridou. 2008. How to say ought in foreign: The composition of weak necessity modals. In Jacqueline Guéron and Jacqueline Lecarme, editors, Studies in Natural Language and Linguistic Theory, volume 75. Springer, Berlin, pages 115–141. find
  55. Wierzbicka, Anna. 1987. The semantics of modality. Folia Linguistica, 21(1):25–43. find
  56. Zaenen, Annie, Lauri Karttunen, and Richard Crouch. 2005. Local textual inference: Can it be defined or circumscribed? In ACL Workshop on Empirical Modelling of Semantic Equivalence and Entailment, pages 31–36, Ann Arbor, MI. find
  57. Zwarts, Frans. 1995. Nonveridical contexts. Linguistic Analysis, 25(3–4):286–312. find