Elamur Research⌕
☰
home / papers / 2017 / Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets

Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets

Paper · 2017 · Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov et al.

Summary

Factuality prediction asks how strongly a source, usually the author, commits to a predicate being true, possible or false. Earlier systems were each built for a single annotated corpus, so results could not be compared across datasets or transferred between them. The authors map FactBank and MEANTIME onto the continuous [−3, +3] scale of the UW corpus with hand-written rules based on each corpus's guidelines. This produces one unified factuality corpus.

They also build a hybrid model. TruthTeller, a rule-based system that uses implicative signatures, gets an expanded predicate lexicon and is run over PropS structures instead of plain dependency trees. Its outputs then serve as features for a linear SVR. Compared with an all-factual baseline, UW features, AMR-based features and rule-based TruthTeller, the hybrid model achieves the best error and correlation on FactBank. It also gives the lowest error on UW and the highest correlation on MEANTIME.

This matters because deep rule-based analysis and supervised learning turn out to complement each other. The unified corpus and the model are publicly released.

Key points

  • Three corpora are unified:
    • FactBank: 77,231 tokens / 3,839 sentences, expert, discrete annotation; only the author's perspective is kept.
    • MEANTIME: 9,743 / 631, expert, discrete annotation.
    • UW: 106,371 / 4,234, crowdsourced, continuous annotation.
  • Example mappings: FactBank CT± becomes ±3, PR± becomes ±2, PS± becomes ±1 and Uu becomes 0. MEANTIME's uncertain possibility becomes ±1.5 and its future possibility ±0.5.
  • The TruthTeller lexicon of about 1,700 predicates gains roughly 800 adjectives, nouns and verbs (about +40%). Their modality status was induced from the subcategorization frames of their German translations in the IMSLex lexicon.
  • Features are the four TruthTeller annotations (signature, clause truth, negation, uncertainty) for three positions: the target, its PropS head and up to 4 children. They are fed to a scikit-learn linear SVR, and evaluation uses MAE and Pearson r.
  • Results of the supervised model (MAE / r):
    • FactBank .59 / .71
    • UW .42 / .66; the UW features alone reach r .71 here.
    • MEANTIME .34 / .47
  • On MEANTIME no system beats the all-factual baseline's MAE of .31, because 90% of its events are labeled +3.

Interesting ideas

  • I1. When merging datasets, map discrete labels onto the continuous scale rather than the reverse, because that direction keeps every original distinction (lossless) and avoids arbitrary discrete-to-discrete choices.
  • I2. A tree abstraction that makes non-verbal predicates (adjectives, conditionals) the direct heads of their arguments lets simple top-down rules reach constructions that plain dependency trees hide.
  • I3. Use the outputs of an explainable rule system as features for a small learned model, which keeps the interpretable signal while letting data correct the rules' blind spots.
  • I4. On heavily skewed data, always report a majority-label baseline and a correlation metric alongside error, since a trivial predictor can win on MAE while capturing no variation.
  • I5. Only propositions judged factual should be admitted into a knowledge store, while hypothetical or negated ones are kept apart.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Apply this when unifying silver/gold data from treebanks or annotation schemes with different label granularities, so no witness's distinctions are destroyed.
  • I2 (tree): A PropS-like post-conversion layer over UD output would let downstream rules treat "was dishonest when he said…" as a predicate chain instead of a leaf under an adverbial clause.
  • I3 (frames): Factuality/implicative-signature tags computed by prag could feed a lightweight regressor whose inputs stay human-readable, preserving explainability.
  • I4 (learn): Add an "everything factual / majority answer" negative control plus Pearson-style checks to evaluation suites where one label dominates.
  • I5 (world): Attach a [−3, +3] commitment score to each event so the world of the text records "might find the truth" or "did not manage" as non-facts rather than as states.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany76 works · 35,008 citations🎓Bar-Ilan UniversityRamat Gan · Israel118 works · 21,077 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan and Iryna Gurevych. 2017. Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2056

@inproceedings{stanovsky2017integrating,
  title = {{Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets}},
  author = {Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan and Iryna Gurevych},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2056},
  url = {https://doi.org/10.18653/v1/p17-2056}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. Meantime, the newsreader multilingual event and time corpus. Proceedings of LREC2016 . find
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking . our card
  3. Malvina Nissim, Paola Pietrandrea, Andrea Sanso, and Caterina Mauri. 2013. Cross-Linguistic Annotation of Modality: a Data-Driven Hierarchical Model. In Proceedings of the 9th Joint ISO - ACL SIGSEM Workshop on Interoperable Semantic Annotation. Potsdam, Germany, pages 7–14. find
  4. Debasish Basak, Srimanta Pal, and Dipak Chandra Patranabis. 2007. Support vector regression. Neural Information Processing-Letters and Reviews 11(10):203–224. find
  5. Judith Eckle-Kohler. 2016. Verbs taking clausal and non-finite arguments as signals of modality revisiting the issue of meaning grounded in syntax. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). Association for Computational Linguistics, volume Volume 1: Long Papers, pages 811–822. find
  6. Tim OGorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. Computing News Storylines page 47. our card
  7. Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing english into abstract meaning representation using syntax-based machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1143– 1154. our card
  8. Arne Fitschen. 2004. Ein Computerlinguistisches Lexikon als komplexes System. PhD Thesis, Universität Stuttgart, Germany. find
  9. Elisa Ghia, Lennart Kloppenburg, Malvina Nissim, Paola Pietrandrea, and Valerio Cervoni. 2016. A construction-centered approach to the annotation of modality. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the 12thISO Workshop on Interoperable Semantic Annotation. European Language Resources Association (ELRA). find
  10. Roser Saurı́ and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. Language resources and evaluation 43(3):227. find
  11. Roser Saurı́ and James Pustejovsky. 2012. Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text. Computational Linguistics 38(2):261–299. our card
  12. Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2016. Getting more out of syntax with props. arXiv preprint . ↗
  13. Valentine Hacquard. 2011. Modality. In Claudia Maienborn, Klaus von Heusinger, and Paul Portner, editors, Semantics: An International Handbook of Natural Language Meaning. HSK 33.2, Berlin: Mouton de Gruyter, pages 1484–1515. find
  14. Lauri Karttunen. 2012. Simple and Phrasal Implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics. Montréal, Canada, pages 124–131. find
  15. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Lisbon, Portugal. find
  16. Amnon Lotan, Asher Stern, and Ido Dagan. 2013. Truthteller: Annotating predicate truth. In HLT-NAACL. pages 752–757. find
  17. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eighth International Conference on Computational Semantics. Tilburg, The Netherlands, IWCS-8 ’09, pages 140–156. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stanovsky-2017-integrating-deep-linguistic-features-factuality/