Elamur Research⌕
☰
home / papers / 2012 / Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text

Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text

Paper · 2012 · Roser Saurí and James Pustejovsky

Summary

The paper addresses how to determine the factual status of events mentioned in text: whether a writer or source presents something as having happened, as not having happened, or as merely possible or probable. The authors treat factuality as a combination of two dimensions, certainty and polarity. They survey the linguistic means that convey it, including negation and modal markers, event-selecting predicates (ESPs) such as implicatives, factives and reporting verbs, subordinating constructions, and discourse-level cues. On this basis they design a linguistically driven computational model whose core algorithm propagates factuality values through nested levels of syntactic embedding. The authors realize this model as De Facto, a system that profiles the factuality of events in English text. It is evaluated on FactBank, a corpus the authors built for this task. De Facto reaches F1 of 0.70 (macro) and 0.80 (micro). The work matters for textual entailment, information extraction, temporal ordering, rumor tracking and analysis of medical records. It also provides lexical and syntactic features that data-driven factuality systems can use.

Key points

  • Factuality values combine certainty (certain, probable, possible) with polarity (positive, negative). This yields categories such as facts, counterfacts, and possibly factual or possibly counterfactual events, always relative to a given source.
  • ESPs (verbs, nouns or adjectives taking event arguments) are central to the model. Examples include manage and know (factual), avoid and prevent (counterfactual), think and speculate (uncertain), and want and offer (underspecified).
  • De Facto was tested on FactBank, which is distributed through the Linguistic Data Consortium. Macro-F1 of 0.70 and micro-F1 of 0.80 are read as lower and upper bounds of its performance.
  • Performance drops on automatically produced dependency trees. Even so, the system still clearly beats the baseline on the harder classes, namely negative polarity and uncertainty.
  • The authors argue the model could be ported to other Romance or Germanic languages, mostly by mapping lexical entries. They also see it as adaptable to domains such as biomedicine by adding hedging markers. Its stated limitations are syntactic coverage and handling ambiguity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brandeis UniversityWaltham · United States30 works · 5,497 citations🔬Barcelona MediaBarcelona · Spain2 works · 945 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Roser Saurí and James Pustejovsky. 2012. Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00096

@article{sauri2012are,
  title = {{Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text}},
  author = {Roser Saurí and James Pustejovsky},
  journal = {Computational Linguistics},
  year = {2012},
  doi = {10.1162/coli\_a\_00096},
  url = {https://doi.org/10.1162/coli_a_00096}
}

References

68 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ACE, 2008. Automatic Content Extraction. English Annotation Guidelines for Relations. Linguistic Data Consortium, version 6.0– 2008.01.07 edition. Available at http://www.ldc.upenn.edu/Projects/ACE/. ↗
  2. Aikhenvald, Alexandra Y. 2004. Evidentiality. Oxford University Press, Oxford. find
  3. Asher, Nicholas. 1993. Reference to Abstract Objects in English. Kluwer Academic Press, Dordrecht. find
  4. Bach, Kent and Robert M. Harnish. 1979. Linguistic Communication and Speech Acts. The MIT Press, Cambridge, MA. find
  5. Biber, Douglas and Edward Finegan. 1989. Styles of stance in English: Lexical and grammatical marking of evidentiality and affect. Text, 9(1):93–124. find
  6. Chafe, Wallace. 1986. Evidentiality in English conversation and academic writing. In W. Chafe and J. Nichols, editors, Evidentiality: The Linguistic Coding of Epistemology. Ablex Publishing Corporation, Norwood, NJ. find
  7. Dalianis, Hercules and Maria Skeppstedt. 2010. Creating and evaluating a consensus for negated and speculative words in a Swedish clinical corpus. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 5–13, Uppsala, Sweden. de Haan, Ferdinand. 1997. The Int… find
  8. Garland, New York. de Marneffe, Marie-Catherine, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC 2006, pages 449–454, Genoa, Italy. our card
  9. Diab, Mona, Bonnie Dorr, Lori Levin, Teruko Mitamura, Rebecca Passonneau, Owen Rambow, and Lance Ramshaw. 2009a. Language Understanding Annotation Corpus. Linguistic Data Consortium, Philadelphia, PA. LDC2009T10. find
  10. Diab, Mona T., Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009b. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop, ACL-IJNLP’09, pages 68–73, Suntec, Singapore. find
  11. Dor, Daniel. 1995. Representations, Attitudes and Factivity Evaluations. An Epistemically-based Analysis of Lexical Selection. Ph.D. thesis, Stanford University. find
  12. Farkas, Richárd, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text. In Proceedings of the 14th CoNLL Conference – Shared Task, pages 1–12, Uppsala, Sweden. our card
  13. Geurts, Bart. 1998. Presuppositions and anaphors in attitude contexts. Linguistics and Philosophy, 21:545–601. find
  14. Givón, Talmy. 1993. English Grammar. A Function-Based Introduction. John Benjamins, Amsterdam. find
  15. Glanzberg, Michael. 2003. Felicity and presupposition triggers. In University of Michigan Workshop in Philosophy and Linguistics, Michigan. find
  16. Halliday, M. A. K. and Christian M. I. M. Matthiessen. 2004. An Introduction to Functional Grammar. Hodder Arnold, London. find
  17. Henriksson, Aron and Sumithra Velupillai. 2010. Levels of certainty in knowledge-intensive corpora: An initial annotation study. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 41–45, Uppsala, Sweden. find
  18. Hickl, Andrew and Jeremy Bensley. 2007. A discourse commitment-based framework for recognizing textual entailment. In Proceedings of the Workshop on Textual Entailment and Paraphrasing, pages 171–176, Prague. find
  19. Hooper, Joan B. 1975. On assertive predicates. In J. Kimball, editor, Syntax and Semantics, IV. Academic Press, New York, pages 91–124. find
  20. Horn, Laurence R. 1972. On the Semantic Properties of Logical Operators in English. Ph.D. thesis, UCLA. Distributed by the Indiana University Linguistics Club, 1976. find
  21. Horn, Laurence R. 1989. A Natural History of Negation. University of Chicago Press, Chicago, IL. find
  22. Hyland, Ken. 1996. Writing without conviction? Hedging in science research articles. Applied Linguistics, 14(4):433–454. find
  23. Karttunen, Lauri. 1970. Implicative verbs. Language, 47:340–358. find
  24. Karttunen, Lauri and Annie Zaenen. 2005. Veridicity. In G. Katz, J. Pustejovsky, and F. Schilder, editors, Dagstuhl Seminar Proceedings, Schloss Dagstuhl, link: http://drops.dagstuhl.de/opus/volltexte/2005/314/pdf/05151. KarttunenLauri.Paper.314.pdf. ↗
  25. Kiefer, Ferenc. 1987. On defining modality. Folia Linguistica, XXI:67–94. find
  26. Kilicoglu, Halil and Sabine Bergler. 2008. Recognizing speculative language in biomedical research articles: A linguistically motivated perspective. BMC Bioinformatics, 9(Suppl 11):S10. find
  27. Kim, Jin-Dong, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 shared task on event extraction. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing: Shared Task, pages 1–9. Boulder, Colorado, USA. our card
  28. Kim, Jin-Dong, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from literature. BMC Bioinformatics, 9(1):10. find
  29. Kiparsky, Paul and Carol Kiparsky. 1970. Fact. In M. Bierwisch and K. E. Heidolph, editors, Progress in Linguistics. A Collection of Papers. Mouton, The Hague, Paris, pages 143–173. find
  30. Kratzer, Angelika. 1991. Modality. In A. van Stechow and D. Wunderlich, editors, Semantik: Ein internationales Handbuch der zeitgenoessischen Forschung. Walter de Gruyter, Berlin, pages 639–650. find
  31. Kudo, Taku and Yuji Matsumoto. 2000. Use of support vector learning for chunk identification. In Proceedings of CoNLL-2000 and LLL-2000, pages 142–144, Lisbon, Portugal. find
  32. Lakoff, George. 1973. Hedges: A study in meaning criteria and the logic of fuzzy concepts. Journal of Philosophical Logic, 2(4):458–508. our card
  33. Light, Marc, Xin Ying Qiu, and Padmini Srinivasan. 2004. The language of bioscience: Facts, speculations, and statements in between. In BioLINK 2004: Linking Biological Literature, Ontologies, and Databases, pages 17–24, Boston, Massachusetts, USA. find
  34. Lyons, John. 1977. Semantics. Cambridge University Press, Cambridge. find
  35. Martin, James R. and Peter R. R. White. 2005. Language of Evaluation: Appraisal in English. find
  36. London and New York: Palgrave Macmillan. find
  37. Medlock, Ben and Ted Briscoe. 2007. Weakly supervised learning for hedge classification in scientific literature. In Proceedings of the 45th ACL, pages 992–999, Prague, Czech Republic. our card
  38. Moilanen, Karo and Stephen Pulman. 2007. Sentiment composition. In Proceedings of the RANLP, pages 27–29, Borovets, Bulgaria. find
  39. Moilanen, Karo, Stephen Pulman, and Yue Zhang. 2010. Packed feelings and ordered sentiments: Sentiment parsing with quasi-compositional polarity sequencing and compression. In Proceedings of the 1st Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2010), pages 36–43… find
  40. Morante, Roser and Walter Daelemans. 2009a. Learning the scope of hedge cues in biomedical texts. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, pages 28–36, Boulder, Colorado, USA. our card
  41. Morante, Roser and Walter Daelemans. 2009b. A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 21–29, Boulder, Colorado, USA. find
  42. Mushin, Ilana. 2001. Evidentiality and Epistemological Stance. John Benjamin, Philadelphia, PA. find
  43. Nairn, Rowan, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Inference in Computational Semantics, ICoS-5, pages 67–76, Buxton, England. our card
  44. Nawaz, Raheel, Paul Thompson, and Sophia Ananiadou. 2010. Evaluating a meta-knowledge annotation scheme for bio-events. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 69–77, Uppsala, Sweden. find
  45. Neviarouskaya, Alena, Helmut Prendinger, and Mitsuru Ishizuka. 2009. Compositionality principle in recognition of fine-grained emotions from text. In Proceedings of the 3rd International ICWSM Conference, pages 278–281, San Jose, California, USA. find
  46. Ohta, Tomoko, Jin-Dong Kim, and Jun’ichi Tsuji. 2007. Guidelines for event annotation. University of Tokyo, link: http://www-tsujii.is.s.u-tokyo.ac.jp/ ∼ genia/release/ Genia event annotation guidelines.pdf. ↗
  47. Özgür, Arzucan and Dragomir Radev. 2009. Detecting speculations and their scopes in scientific text. In Proceedings of the 2009 EMNLP Conference, pages 1398–1407, Suntec, Singapore. find
  48. Palmer, Frank R. 1986. Mood and Modality. Cambridge University Press, Cambridge. find
  49. Polanyi, Livia and Annie Zaenen. 2006. Contextual valence shifters. In W. B. Croft, J. Shanahan, Y. Qu, and J. Wiebe, editors, Computing Attitude and Affect in Text: Theory and Applications, volume 20. Springer-Verlag, New York, pages 1–10. find
  50. Prabhakaran, Vinodkumar, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In CoLing 2010. Poster Volume, pages 1014–1022, Beijing, China. our card
  51. Prasad, Rashmi, Nikhil Dinesh, Alan Lee, Aravind Joshi, and Bonnie Webber. 2007. Attribution and its annotation in the Penn Discourse Treebank. Traitement Automatique des Langues, 47(2):43–64. find
  52. Pustejovsky, James, Bob Knippen, Jessica Littman, and Roser Saurı́. 2005. Temporal and event information in natural language text. Language Resources and Evaluation, 39(2):123–164. find
  53. Pustejovsky, James, Marc Verhagen, Roser Saurı́, Jessica Littman, Robert Gaizauskas, Graham Katz, Inderjeet Mani, Robert Knippen, and Andrea Setzer. 2006. TimeBank 1.2. Linguistic Data Consortium, Philadelphia, PA. LDC2006T08. find
  54. Rizomilioti, Vassiliki. 2006. Exploring epistemic modality in academic discourse using corpora. In E. Arnó Macià, A. Soler Cervera, and C. Rueda Ramos, editors, Information Technology in Languages for Specific Purposes, volume 7. Springer, Berlin, pages 53–71. find
  55. Rubin, Victoria L. 2006. Identifying Certainty in Texts. Ph.D. thesis, Syracuse University. find
  56. Rubin, Victoria L. 2007. Stating with certainty or stating with doubt: Intercoder reliability results for manual annotation of epistemically modalized statements. In Proceedings of the NAACL-HLT 2007, pages 141–144, Rochester, NY. find
  57. Rubin, Victoria L. 2010. Epistemic modality: From uncertainty to certainty in the context of information seeking as interactions with texts. Information Processing and Management, 46:533–540. find
  58. Saurı́, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. thesis, Brandeis University. find
  59. Saurı́, Roser and James Pustejovsky. 2007. Determining modality and factuality for text entailment. In Proceedings of the First IEEE International Conference on Semantic Computing, pages 509–516, Irvine, California, USA. find
  60. Saurı́, Roser and James Pustejovsky. 2009a. FactBank 1.0. Linguistic Data Consortium, Philadelphia, PA. LDC2009T23. find
  61. Saurı́, Roser and James Pustejovsky. 2009b. FactBank. A corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. find
  62. Saurı́, Roser, Marc Verhagen, and James Pustejovsky. 2006. SlinkET: A partial modal parser for events. In Proceedings of LREC 2006, pages 1332–1337, Genoa, Italy. find
  63. Shatkay, Hagit, Fengxia Pang, Andrey Rzhetsky, and W. John Wilbur. 2008. Multi-dimensional classification of biomedical text: Toward automated, practical provision of high-utility text to diverse users. Bioinformatics, 24:2086–2093. find
  64. Szarvas, Gyorgy. 2008. Hedge classification in biomedical texts with a weakly supervised selection of keywords. In ACL 08: HLT, pages 281–289, Columbus, Ohio, USA. van Valin, Robert D. and Randy J. LaPolla. 1997. Syntax. Structure, Meaning and Function. Cambridge University Press, Cambridge. our card
  65. Velldal, Erik, Lilja Ovrelid, and Stephan Oepen. 2010. Resolving speculation: Maxent cue classification and dependency-based scope rules. In Proceedings of the 14th CoNLL: Shared Task, pages 48–55, Uppsala, Sweden. find
  66. Vincze, Veronika, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The BioScope corpus: Biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics, 9(Suppl 11):S9. find
  67. Wiebe, Janyce, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210. find
  68. Wilbur, W. John, Andrey Rzhetsky, and Hagit Shatkay. 2006. New directions in biomedical text annotation: definitions, guidelines and corpus construction. BMC Bioinformatics, 7(1):356–365. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sauri-2012-are-you-sure-that-this/