Elamur Research⌕
☰
home / papers / 2014 / A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards

A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards

Summary

Work on relation extraction and semantic parsing depends on several hand-annotated resources. These were built under different guidelines and for different purposes, which makes them hard to reuse together. This paper, a collaboration between the JHU HLTCOE and the Linguistic Data Consortium, surveys four standards side by side: ACE, its simplified successor ERE, TAC-KBP Slot-filling, and FrameNet.

The authors first show how entity-level decisions shape relation and event annotation. These decisions include type inventories, subtypes, entity classes and mention extents. They then contrast ACE and ERE on relation types, argument constraints, modality, tense, syntactic classes and triggers. Next, they outline the narrow, question-oriented TAC-KBP slot schema. Finally, they map relations, events and attributes onto FrameNet frames.

The main conclusions are:

  • ERE gives up some of ACE's granularity to gain annotator consistency.
  • TAC-KBP targets entity slots and does not model events explicitly.
  • FrameNet is the most expressive scheme. Content from the other standards can generally be mapped into it, provided type and argument granularity are adjusted.

The survey helps practitioners align, compare and convert annotated corpora.

Key points

  • ACE was created by NIST in 1999, and its last evaluation took place in 2008. ERE was designed under DARPA DEFT as a lighter variant with these changes:
    • Facility and Location are merged into Location.
    • Vehicle and Weapon are dropped, and a Title type is added.
    • Subtypes are removed.
    • Only Specific mentions are tagged.
  • ERE annotates only asserted relations. ACE also covers hypothetical ones through Modality (ASSERTED/OTHER) and four Tense values. ACE additionally uses Syntactic Classes and clause-level spans, whereas ERE uses an optional minimal trigger.
  • The relation inventories differ:
    • ERE adds Physical.Origin, Social.Membership and Social.Role.
    • ERE lacks ACE's Near and Part-Whole.Geographical subtypes.
    • ERE has only two Affiliation subtypes, while ACE has many.
  • FrameNet 1.5 contains 1020 frames and 11830 lexical units, and mappings to it can be one-to-many:
    • ERE Affiliation.Employment/Membership corresponds to both Being Employed and Membership.
    • The Kinship frame corresponds to five TAC-KBP slots.
    • The core elements of the Extradition frame match ERE's Justice-Extradition arguments exactly. ACE's extra Crime and Time arguments map to non-core elements.
  • The TAC-KBP Cold Start task, introduced in 2012, builds knowledge bases from collections of roughly O(50,000) documents. These KBs are evaluated by sampling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

46 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 44 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-2907

@inproceedings{aguilar2014comparison,
  title = {{A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards}},
  author = {Aguilar, Jacqueline and Beller, Charley and McNamee, Paul and Durme, Benjamin Van and Strassel, Stephanie and Song, Zhiyi and Ellis, Joe},
  booktitle = {Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation},
  year = {2014},
  doi = {10.3115/v1/w14-2907},
  url = {https://doi.org/10.3115/v1/w14-2907}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. find our card
  2. Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. Semeval-2007 task 19: Frame semantic structure extraction. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 99–104, Prague, Czech Republic, June. Association for Computational Linguistics. find our card
  3. Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A Smith. 2010. Probabilistic frame-semantic parsing. In Proceedings of NAACL-HLT, pages 948– 956. Association for Computational Linguistics. find our card
  4. George Doddington, Alexis Mitchell, Mark Przybocki, Lancec Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ace) program- tasks, data, and evaluation. In Proceedings of LREC 2004: Fourth International Conference on Language Resources and Evaluation, Lisbon,… find our card
  5. Charles J Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20–32. find
  6. Charles Fillmore. 1982. Frame semantics. In Linguistics in the morning calm, pages 111–137. Hanshin Publishing Co. find
  7. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational linguistics, 28(3):245–288. find our card
  8. Linguistic Data Consortium. 2005. ACE (automatic content extraction) English annotation guidelines for events. https://www.ldc.upenn.edu/collaborations/past-projects/ace. Version 5.4.3 2005.07.01. ↗
  9. Linguistic Data Consortium. 2006. ACE (automatic content extraction) English annotation guidelines for entities. https://www.ldc.upenn.edu/collaborations/past-projects/ace, Version 5.6.6 2006.08.01. ↗
  10. Linguistic Data Consortium. 2008. ACE (automatic content extraction) English annotation guidelines for relations. https://www.ldc.upenn.edu/collaborations/past-projects/ace. Version 6.0 2008.01.07. ↗
  11. Linguistic Data Consortium. 2013a. DEFT ERE annotation guidelines: Entities v1.1, 05.17.2013. find
  12. Linguistic Data Consortium. 2013b. DEFT ERE annotation guidelines: Events v1.1. 05.17.2013. find
  13. Linguistic Data Consortium. 2013c. DEFT ERE annotation guidelines: Relations v1.1. 05.17.2013. find
  14. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Rada Mihalcea and Phil Edmonds, editors, Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 9–12, Barcelona, Spain, July. Association for Computational Linguisti… find
  15. Paul McNamee, Hoa Trang Dang, Heather Simpson, Patrick Schone, and Stephanie Strassel. 2010. An evaluation of technologies for knowledge base population. In Proceedings of LREC. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aguilar-2014-comparison-events-relations-across-ace/