Elamur Research⌕
☰
home / papers / 2015 / From Light to Rich ERE: Annotation of Entities, Relations, and Events

From Light to Rich ERE: Annotation of Entities, Relations, and Events

Summary

The DARPA DEFT program needed annotated data that could serve many component technologies built around one goal: populating a knowledge base with entities, events and the links between them. The data had to cover several languages and genres, including informal discussion forums. LDC responded with the ERE (Entities, Relations and Events) task and rolled it out in phases. Light ERE, a simplified derivative of ACE, narrowed what could be tagged so that consistent data could be produced quickly in English, Chinese and Spanish. Rich ERE then broadens the scope:

  • nonspecific entities and nominal head marking
  • a separate Facility type
  • argument fillers
  • a relation inventory aligned with TAC KBP Slot Filling
  • realis and specificity labels
  • event hoppers, a looser form of event coreference that tolerates differences in mention and argument granularity

The paper also describes released and planned corpora, a way to prioritise event-rich documents, and the planned inter-annotator agreement methodology. The work matters because it supplies training data matching the TAC KBP event evaluations. It also lays the groundwork for event-event relations and for cross-document and cross-lingual event representation.

Key points

  • Light ERE restricts annotation compared with ACE:
    • only specific entities and actual relations are tagged;
    • entity subtypes are dropped;
    • only attested actual events are labelled, with no irrealis events or arguments;
    • events are coreferenced within a document only.
  • Rich ERE adds several entity-level changes:
    • NonSPC covers both generic and underspecified entities;
    • heads are marked manually, but only for nominal mentions;
    • Location is split into Location and Facility (man-made structures count as Facility);
    • weapons, vehicles, commodities and titles become argument fillers rather than entities.
  • Rich ERE doubles the relation ontology from ten subtypes to twenty, following TAC KBP Slot Filling. Three of the new subtypes take fillers: personalsocial-role (Title), generalaffiliation-orgwebsite (URL) and generalaffiliation-personage (Age).
  • Released Light ERE data:
    • about 570K words of English (NW and DF);
    • 200K words of Chinese DF;
    • roughly 100K words of Spanish in progress;
    • a Chinese-English parallel corpus of about 100K words, parts of which also carry CTB and ECTB treebank layers.
  • Rich ERE data: 32,420 English words (91 documents) were finished at the time of writing. The phase target is 400Kw per language for English, Chinese and Spanish, with 100Kw each of Chinese and Spanish parallel to English translations.

Interesting ideas

  • I1. Start light, then enrich. Releasing a deliberately simple scheme first lets data production scale fast. Complexity is added only once system needs are clearer, which is a sensible approach for any new multilingual annotation project with an uncertain scope.
  • I2. Choose documents by automatic trigger density. Documents were ranked by event triggers per 1,000 tokens, using a neural tagger trained on ACE 2005 with word2vec features. This kept annotators away from content-poor texts, and the same pre-ranking could cut wasted effort for any sparse phenomenon.
  • I3. Use a looser coreference unit. Event hoppers group mentions that clearly describe the same happening even when they fall short of strict identity. This fits how people actually describe events in forums and across documents, and avoids forcing annotators into brittle decisions.
  • I4. Model marginal participants as fillers, not entities. Items like titles, URLs, ages and weapons are annotated only when they fill a role in a tagged relation or event. This keeps the entity layer lean while still supporting slot-filling-style knowledge base needs.
  • I5. Align layers on the same text and match agreement hierarchically. Putting ERE and treebank annotation on the same text enables joint modelling across layers. Scoring agreement level by level (entity mentions matched by span, relations and events matched through their arguments) is a reusable design for any nested annotation scheme.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

114 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 105 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w15-0812

@inproceedings{song2015light,
  title = {{From Light to Rich ERE: Annotation of Entities, Relations, and Events}},
  author = {Song, Zhiyi and Bies, Ann and Strassel, Stephanie and Riese, Tom and Mott, Justin and Ellis, Joe and Wright, Jonathan and Kulick, Seth and Ryant, Neville and Ma, Xiaoyi},
  booktitle = {Proceedings of the 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation},
  year = {2015},
  doi = {10.3115/v1/w15-0812},
  url = {https://doi.org/10.3115/v1/w15-0812}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, Joe Ellis. 2014. A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards. find our card
  2. ACL 2014: 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, June 22-27. 2nd Workshop on Events: Definition, Detection, Coreference, and Representation. find
  3. Ann Bies, Justin Mott, Seth Kulick, Jennifer Garland, Colin Warner. 2014. Incorporating Alternate Translations into English Translation Treebank. In Proceedings of the 9th Edition of the Language Resources and Evaluation Conference (LREC 2014), Reykjavik, May 26-31. find
  4. Branimir Boguraev and Rie Kubota Ando. 2005. TimeML-Compliant Text Analysis for Temporal Reasoning. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pp. 997--1003. find
  5. DARPA. 2012. Broad Agency Announcement: Deep Exploration and Filtering of Text (DEFT). Defense Advanced Research Projects Agency, DARPA-BAA-12-47. find
  6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program- tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), Lisbon,… find our card
  7. Joe Ellis, Jeremy Getman, Stephanie M. Strassel. 2014. Overview of Linguistic Resources for the TAC KBP 2014 Evaluations: Planning, Execution, and Results. In Proceedings of TAC KBP 2014 Workshop, National Institute of Standards and Technology, Gaithersburg, Maryland, USA, November 17-18, 2014. find
  8. Seth Kulick, Ann Bies, Justin Mott. 2014. Inter-annotator Agreement for ERE Annotation. find
  9. ACL 2014: 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, June 22-27. 2nd Workshop on EVENTS: Definition, Detection, Coreference, and Representation. find
  10. Linguistic Data Consortium. 2005. ACE (Automatic Content Extraction) English Annotation Guidelines for Events Version 5.4.3. find
  11. William C. Mann and Sandra Thompson. 1988. Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. Text 8(3) (243-281). find our card
  12. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR, 2013. find
  13. Teruko Mitamura, Yukari Yamakawa, Sue Holm, Zhiyi Song, Ann Bies, Seth Kulick, Stephanie Strassel. 2015. find
  14. Event Nugget Annotation: Processes and Issues. The 2015 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL HLT 2015). 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation. find
  15. Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, Ralph Weischedel. 2007. OntoNotes: A Unified Relational Semantic Representation. In Proceedings of the First IEEE International Conference on Semantic Computing (ICSC-07). Irvine, CA. find
  16. Rashmi Prasad, Bonnie Webber, Aravind Joshi. 2014. Reflections on the Penn Discourse TreeBank, Comparable Corpora and Complementary Annotation. Computational Linguistics, Vol. 40, No. 4, pp. 921-950, doi:10.1162/COLI_a_00204. find our card
  17. James Pustejovsky, and Amber Stubbs. 2011. Increasing Informativeness in Temporal Annotation. Linguistic Annotation Workshop V (LAW-V), Portland, Oregon. find our card
  18. Heather Simpson, Christopher Cieri, Kazuaki Maeda, Kathryn Baker, Boyan Onyshkevych. 2008. Human Language Technology Resources for Less Commonly Taught Languages: Lessons Learned Toward Creation of Basic Language Resources. In Proceedings of SALTMIL Workshop: Free/Open-Source Language Resources for… find
  19. Stephanie Strassel, Dan Adams, Henry Goldberg, Jonathan Herr, Ron Keesing, Daniel Oblinger, Heather Simpson, Robert Schrag, Jonathan Wright. 2010. The DARPA Machine Reading Program - Encouraging Linguistic and Reasoning Research with a Series of Reading Tasks. In Proceedings of the 7th International… find
  20. Christopher Walker, Stephanie Strassel, Julie Medero, Kazuaki Maeda. 2006. find
  21. ACE 2005 Multilingual Training Corpus. Linguistic Data Consortium, LDC Catalog No.: LDC2006T06. find
  22. Jonathan Wright, Kira Griffitt, Joe Ellis, Stephanie Strassel, Brendan Callahan. 2012. Annotation Trees: LDC's Customizable, Extensible, Scalable Annotation Infrastructure. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC 2012), Istanbul, May 21-27. find
  23. Xiuhong Zhang and Nianwen Xue. 2012. Extending and Scaling up the Chinese Treebank Annotation. In Proceedings of the 2nd CIPS-SIGHAN Joint Conference on Chinese Language Processing (CLP-2012), Tianjin, China. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2015-light-rich-ere-annotation-entities/