Elamur Research⌕
☰
home / papers / 2023 / Towards Generative Event Factuality Prediction

Towards Generative Event Factuality Prediction

Paper · 2023 · John Murzaku, Tyler Osborne, Amittai Aviram et al.

Summary

Most work on event factuality asks only how the author presents an event: as certain, possible, unknown, or not happened. This paper extends the task to factuality attributed to sources mentioned inside the text, such as a person who "said" something. The authors use FactBank, an expert-annotated newswire corpus and the only one with nested-source annotation. They build a database representation of it and derive two tasks: source-and-target factuality (STF) and author-only factuality (AOF).

Each judgment is encoded as a triple of source head word, target event head word and one of five factuality values. Flan-T5-base is fine-tuned to generate these structures end to end, without gold event heads. The authors compare output formats, such as tuple/triplet strings against in-line annotation of the sentence. They also test task prefixes and multi-task mixtures with other corpora.

Careful output formatting and suitable multi-task mixtures both help. The system sets new state-of-the-art end-to-end results on FactBank author-only factuality and gives the first results for source-and-target factuality. This matters because a reader must separate what the author asserts from what others are reported to believe.

Key points

  • Factuality labels are reduced to five values: CT+, CT-, PR+, PR- and UU. The FactBank distinction between probable and possible is merged to reduce data fragmentation.
  • A judgment is a (source, target, value) triple over syntactic head words, with a special AUTHOR token for the writer. The nested chain of sources (Author → Mary → John) is collapsed to the last source only.
  • The model is Flan-T5-base. Tuple/triplet output beats in-line annotation ("Anno") in exact-match evaluation, mainly because Anno does worse on the frequent CT+ label.
  • On FactBank author-only, the best exact-match F1 is 0.897, from multi-task learning with the FBST data (FBAO*, FBST). The plain FBAO setting reaches 0.866. AOF scores about 13.3 points absolute above STF.
  • Other author-only corpora are surveyed and used for combinations: LU, UW, LDCCB, MEANTIME, MegaVeridicality, UDS-IH2, CommitmentBank and RP.

Interesting ideas

  • I1. Representing "who presents which event as how factual" as a small triple anchored on head words gives a compact, checkable target that could be reused for any attribution-style annotation.
  • I2. The authors merge hard-to-separate categories (probable vs. possible), trading granularity for more reliable labels and less sparse data, which suits small corpora.
  • I3. Choosing the output format of a generative model is treated as a design variable, and structured tuples beat in-line markup on the frequent cases.
  • I4. The same system is reported under both a macro-average and an exact-match micro metric, which reveals different strengths (Anno helps rare labels but hurts overall).
  • I5. Keeping the author's commitment separate from a reported source's commitment avoids treating reported claims as facts in the text's world.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could attach a factuality triple (AUTHOR or speaker-entity, event head, CT+/PR+/UU…) to every frame produced under verbs like say or believe.
  • I2 (world): world states could carry a coarse certainty scale (certain/possible/unknown × polarity) instead of fine grades that are hard to justify.
  • I3 (learn): when the LLM teacher produces silver labels, asking for explicit tuples rather than annotated sentences may give cleaner, easier-to-verify supervision.
  • I4 (learn): evaluations could routinely report both a per-label macro score and an exact-match score, so that gains on rare classes do not hide losses on common ones.
  • I5 (answer): QA on fables and tales could answer "did X happen?" differently from "did the fox say X happened?", using the source in the stored triple.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States43 works · 2,571 citations🎓Boston UniversityBoston · United States26 works · 4,022 citations🎓University of Massachusetts BostonBoston · United States3 works · 300 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as John Murzaku, Tyler Osborne, Amittai Aviram and Owen Rambow. 2023. Towards Generative Event Factuality Prediction. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.44

@inproceedings{murzaku2023generative,
  title = {{Towards Generative Event Factuality Prediction}},
  author = {John Murzaku, Tyler Osborne, Amittai Aviram and Owen Rambow},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.44},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.44}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3528–3539, Brussels, Belgium. Association for Computational Linguistics.
  2. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. ↗
  3. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107–124. ↗
  4. Lingjia Deng and Janyce Wiebe. 2015. MPQA 3.0: An entity/event-level sentiment corpus. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1323–1328, Denver, Colorado. Association for Computational Linguistics. find
  5. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 68–73, Suntec, Singapore. Association for Computational Linguistics. ↗
  6. William Falcon et al. 2019. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, 3(6). ↗
  7. Tianhao Gao, Jun Fang, Hanyu Liu, Zhiyuan Liu, Chao Liu, Pengzhang Liu, Yongjun Bao, and Weipeng Yan. 2022. LEGO-ABSA: A prompt-based task assemblable unified generative framework for multi-task aspect-based sentiment analysis. In Proceedings of the 29th International Conference on Computational Linguistics, pages 7002–7012, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  8. Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails on pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097. ↗
  9. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal. Association for Computational Linguistics. ↗
  10. Amnon Lotan, Asher Stern, and Ido Dagan. 2013. TruthTeller: Annotating predicate truth. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 752– 757, Atlanta, Georgia. Association for Computational Linguistics. find
  11. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begoña Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the NewsReader multilingual event and time corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4417–4422, Portorož, Slovenia. European Language Resources Association (ELRA). find
  12. John Murzaku, Peter Zeng, Magdalena Markowska, and Owen Rambow. 2022. Re-examining FactBank: Predicting the author’s presentation of factuality. In Proceedings of the 29th International Conference on Computational Linguistics, pages 786–796, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  13. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of the Fifth International Workshop on Inference in Computational Semantics (ICoS-5).
  14. Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8600–8607. find
  15. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4393–4399, Florence, Italy. Association for Computational Linguistics. ↗
  16. Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg, Owen Rambow, Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gregory Werner, Yorick Wilks, and Janyce Wiebe. 2015. A new dataset and evaluation for belief/factuality. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 82–91, Denver, Colorado. Association for Computational Linguistics. find
  17. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China. Coling 2010 Organizing Committee.
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  19. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics. ↗
  20. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  21. Roser Saurí and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. Language resources and evaluation, 43(3):227–268. ↗
  22. Roser Saurí and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational Linguistics, 38(2):261– 299. ↗
  23. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357, Vancouver, Canada. Association for Computational Linguistics. ↗
  24. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. ↗
  25. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005a. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2/3):164–210. ↗
  26. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005b. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165–210. ↗
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  28. Qingrong Xia, Bo Zhang, Rui Wang, Zhenghua Li, Yue Zhang, Fei Huang, Luo Si, and Min Zhang. 2021. A unified span-based approach for opinion mining with syntactic constituents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1795–1804, Online. Association for Computational Linguistics. find
  29. Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021. Towards generative aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 504–510, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/murzaku-2023-generative-event-factuality-prediction/