Elamur Research⌕
☰
home / papers / 2008 / FATE: a FrameNet-Annotated Corpus for Textual Entailment

FATE: a FrameNet-Annotated Corpus for Textual Entailment

Summary

Earlier studies suggested that predicate-argument structure captures many phenomena in textual entailment data, such as paraphrases and argument alternations. However, the only FrameNet-based RTE system had performed only moderately. The authors give three possible reasons: FrameNet may not cover enough of the data, frame semantic parsers may be unreliable, or systems may use frame information poorly. To help tell these explanations apart, they built FATE, a manually produced gold-standard corpus. It covers all 800 text–hypothesis pairs of the RTE-2 test set, annotated with frames and semantic roles. The annotation uses a new scheme. It is full-text annotation, but only of the "relevant" frame evoking elements rather than every word that could evoke a frame. Consistency was checked through inter- and intra-annotator agreement and weekly supervision meetings. The resulting statistics show that very few instances needed an Unknown frame. This suggests FrameNet coverage is probably not what holds back frame-based RTE. The corpus can also serve as a benchmark for parsers such as Shalmaneser and as a clean testbed for designing entailment systems.

Key points

  • FATE annotates the 800 (T, H) pairs of the RTE-2 test set with FrameNet frames and roles. It is distributed in SALSA/TIGER XML format.
  • Annotation took 230 hours in total. Positive pairs averaged about 13 minutes each and negative pairs about 21 minutes.
  • The corpus contains 4,489 frames (about 5.6 per pair) and 9,518 roles (about 2.1 per frame).
  • Unknown frames make up only 8% of frame instances (373 cases) and unknown roles only 1%. This contrasts with SALSA's German annotation, where about a third of verb occurrences lacked a suitable frame.
  • Intra-annotator agreement was measured on 109 repeated sentence pairs. It reached 97% for FEEs, 98% for frames and 88% for roles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Treebanks · Semantic role labeling

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{burchardt2008fate,
  title = {{FATE: a FrameNet-Annotated Corpus for Textual Entailment}},
  author = {Burchardt, Aljoscha and Pennacchiotti, Marco},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/37cq5v73ysjz},
  url = {https://doi.org/10.63317/37cq5v73ysjz}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of COLING-ACL, Montreal, Canada. find our card
  2. Roy Bar-Haim, Idan Szpektor, and Oren Glickman. 2005. Definition and analysis of intermediate entailment levels. In Definition and Analysis of Intermediate Entailment Levels, Ann Arbor, Michigan. find
  3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor, editors. 2006. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  4. Johan Bos and Katja Markert. 2005. Combining shallow and deep nlp methods for recognizing textual entailment. In Pascal, Proceedings of the First Challenge Workshop, Recognizing Textual Entailment., Southampton, UK. find
  5. Aljoscha Burchardt and Anette Frank. 2006. Approximating Textual Entailment with LFG and FrameNet Frames. In Proceedings of PASCAL RTE2 Workshop. find
  6. Aljoscha Burchardt, Katrin Erk, and Anette Frank. 2005. A WordNet Detour to FrameNet. In B. Fisseni, H.-C. Schmitz, B. Schröder, and P. Wagner, editors, Sprachtechnologie, mobile Kommunikation und linguistische Resourcen, volume 8 of Computer Studies in Language and Speech. Peter Lang, Frankfurt/Main. find
  7. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006a. The salsa corpus: a german corpus resource for lexical semantics. In Proceedings of LREC 2006, Genoa, Italy. find our card
  8. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006b. Salto – a versatile multi-level annotation tool. In Proceedings of LREC 2006, Genoa, Italy. find
  9. Aljoscha Burchardt, Nils Reiter, Stefan Thater, and Anette Frank. 2007. A semantic approach to textual entailment: System evaluation and task analysis. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague. find
  10. M. Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. find our card
  11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence D’Alché-Buc, editors, Evaluating Predictive Uncertainty, Visual Object Categorization and Textual Entailment, volume 3944 of Lecture Notes in Computer Science, pages 1–27, Heidelberg, Germany. Springer. find
  12. Katrin Erk and Sebastian Pado. 2006. Shalmaneser - a flexible toolbox for semantic role assignment. In Proceedings of LREC 2006, Genoa, Italy. find
  13. K. Garoufi. 2007. Towards a better understanding of applied textual entailment: Annotation and evaluation of the rte-2 dataset. M.Sc. thesis, Saarland University. find
  14. Paul Kingsbury, Martha Palmer, and Mitch Marcus. 2002. Adding semantic annotation to the Penn TreeBank. In Proceedings of the Human Language Technology Conference, San Diego. find
  15. Ken Litkowski. 2006. Componential analysis for recognizing textual entailment. In Proceedings of PASCAL RTE2 Workshop. find
  16. S. Pado. 2007. Cross-Lingual Annotation Projection Models for Role-Semantic Information. Ph.D. thesis, Saarland University, Germany. find
  17. Josef Ruppenhofer, Michael Ellsworth, Miriam R. L. Petruck, and Christopher R. Johnson. 2007. Framenet: Theory and practice. Available at http://framenet.icsi.berkeley.edu/. ↗
  18. Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors. 2007. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, Prague, June. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/burchardt-2008-fate-framenet-annotated-corpus-textual/