Elamur Research⌕
☰
home / papers / 2022 / Re-Examining FactBank: Predicting the Author's Presentation of Factuality

Re-Examining FactBank: Predicting the Author's Presentation of Factuality

Paper · 2022 · John Murzaku, Peter Zeng, Magdalena Markowska et al.

Summary

The paper studies factuality prediction: deciding whether an author presents an event as certain, probable, possible or unspecified, and with which polarity. The authors focus on FactBank. They found a systematic flaw in the widely used CoNLL-style projection of FactBank by Stanovsky et al. (2017). That projection sometimes took a label belonging to a mentioned agent's perspective when it should have used the author's own view (UU). It also mixed sentences from the same article across train and test. They fixed the labels and re-split the data by article, which means earlier FactBank numbers no longer hold. They then reran multi-task and combined-corpus experiments and proposed categorical F-measure alongside Pearson r and MAE. The paper adds no new architecture. Instead, careful data work and well-chosen corpus combinations, chiefly FactBank with CB and MegaVeridicality, give new state-of-the-art results on FactBank, CB, MV and UW. Error analyses explain which corpora help each other and which ones clash. The main lesson is that cleaning data and checking compatibility can matter more than model complexity.

Key points

  • The corrected FactBank keeps only the author's perspective, using seven labels mapped to −3…+3 (CT-, PR-, PS-, UU, PS+, PR+, CT+). Splits are by article at roughly 70-20-10. On the corrected data, Pearson r falls by about 2% and MAE rises by about 10% compared with the old version.
  • Training on FB+CB+MV with multi-task learning gives r 0.913 and MAE 0.536 on CB, about a 21% error reduction on both measures over the previous best.
  • A hybrid system routes each event head by its spaCy POS tag. Nouns go to the FactBank-only model and other heads go to the FB+CB model. This gives the best 5-way classification, cutting macro-averaged error by 12%, and has the lowest variance.
  • On LDCCB (forum posts), a model trained on LDC reaches r 0.822 and MAE 0.361. A FactBank-trained model gets only r 0.616 and MAE 0.630, which points to genre incompatibility.
  • In 100 sampled FactBank→LDC errors, half trace back to faulty gold labels. Unclear text accounts for 13, embedded clauses for 12 and hypotheticals for 11.

Interesting ideas

  • I1. Before trusting a popular derived dataset, compare it with the original annotation scheme, since a single mapping slip can quietly invalidate years of reported results.
  • I2. Assign whole documents to a single split so that shared topics and phrasings cannot leak from train into test.
  • I3. Ordinal labels such as certainty levels should also be scored categorically. Correlation treats a PR+/PS+ confusion the same as a PR+/CT+ confusion, but downstream uses care about the difference.
  • I4. Route each item to the specialist model whose training corpus actually covers its construction, here by the POS of the event head.
  • I5. Track who is committed to an event separately for each source, so that a reported agent's certainty is never mistaken for the author's.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Audit every converted treebank and silver set against its source, for example dialect conversions or LLM-labelled data, by sampling items and re-deriving their labels from the originals.
  • I2 (learn): Make document-level splitting the default for all evaluation sets, and report how much scores drop compared with sentence-level splits as a leakage control.
  • I3 (frames): Evaluate the speech-act and hedging layer of prag with per-class F1 so that confusing "probable" with "certain" is visible, not hidden in an averaged score.
  • I4 (frames): Choose factuality or commitment rules by the UD category of the frame's head, with noun events and clausal-complement verbs handled separately, and keep the routing decision in the explanation trace.
  • I5 (world): Store each world-state fact with its belief source and commitment level, so that "X denied Y happened" records Y as X's claim while the narrator's stance stays unspecified.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States27 works · 2,057 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as John Murzaku, Peter Zeng, Magdalena Markowska and Owen Rambow. 2022. Re-Examining FactBank: Predicting the Author's Presentation of Factuality. Proceedings of the 29th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{murzaku2022re,
  title = {{Re-Examining FactBank: Predicting the Author's Presentation of Factuality}},
  author = {John Murzaku, Peter Zeng, Magdalena Markowska and Owen Rambow},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/jner-0188},
  url = {https://doi.org/10.48448/jner-0188}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107–124. ↗
  2. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 68–73, Suntec, Singapore. Association for Computational Linguistics. ↗
  3. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640. ↗
  4. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  5. Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails on pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097. ↗
  6. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal. Association for Computational Linguistics. ↗
  7. Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg, Owen Rambow, Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gregory Werner, Yorick Wilks, and Janyce Wiebe. 2015. A new dataset and evaluation for belief/factuality. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 82–91, Denver, Colorado. Association for Computational Linguistics. find
  8. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China. Coling 2010 Organizing Committee.
  9. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics.
  10. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  11. Roser Saurí and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. ↗
  12. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In ACL.
  13. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  14. Amnon Lotan, Asher Stern, and Ido Dagan. 2013. TruthTeller: Annotating predicate truth. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 752– 757, Atlanta, Georgia. Association for Computational Linguistics. find
  15. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begoña Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the NewsReader multilingual event and time corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4417–4422, Portorož, Slovenia. European Language Resources Association (ELRA). find
  16. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of the fifth international workshop on inference in computational semantics (icos-5).
  17. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. arXiv preprint arXiv:1907.03227. ↗
  18. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phil Yeres, Jason Phang, Haokun Liu, Phu Mon Htut, , Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019. jiant 1.3: A software toolkit for research on general-purpose text understanding models. http://jiant.info/. ↗
  19. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. ↗
  20. Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. 2021. Delving into deep imbalanced regression. In International Conference on Machine Learning (ICML). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/murzaku-2022-re-examining-factbank-predicting-author/