Elamur Research⌕
☰
home / papers / 2009 / Overview of BioNLP'09 Shared Task on Event Extraction

Overview of BioNLP'09 Shared Task on Event Extraction

Summary

The BioNLP'09 Shared Task moved biomedical information extraction past simple protein-protein interaction (PPI) pairs toward detailed bio-molecular events. These events can nest, for example a regulation whose theme is another event, and that richer structure is what pathway and GOA-style databases need. The organizers built the data from the GENIA event corpus. They added GENETAG-style gene-or-gene-product (GGP) annotation, restricted primary non-event arguments to proteins, and marked equivalent entity names so that alternative valid answers count as correct. Gold protein mentions were provided, so participants could concentrate on event extraction. The challenge had three subtasks: core event detection (mandatory), secondary-argument enrichment, and negation/speculation detection. It ran for 12 weeks; 42 teams showed initial interest and 24 submitted final results. Simple events were extracted with F-scores approaching 70%. Binding, regulation and Task 3 remained much harder. Combining the top systems in an ensemble gave about a 4-point gain. The task set up a benchmark and resource for fine-grained bio-event extraction.

Key points

  • Data splits (abstracts / sentences / words / events, Task 1 counts):
    • Training: 800 / 7,449 / 176,146 / 8,597
    • Development: 150 / 1,450 / 33,937 / 1,809
    • Test: 260 / 2,447 / 57,367 / 3,182, taken from an unpublished part of the GENIA event corpus
  • Event types and arguments: the task covers nine event types. Three are regulation types (Regulation, Positive regulation and Negative regulation), and the other six are Binding, Localization, Phosphorylation, Protein catabolism, Transcription and Gene expression.
    • Primary arguments are Theme and Cause.
    • Secondary arguments are Site, CSite, AtLoc and ToLoc.
  • Task 1 F-score ranges:
    • Simple events: roughly 31–70%
    • Binding: 5–44%
    • Regulation: 1–40%
  • Task 2 and Task 3 results:
    • The best overall Task 2 result was UT+DBCLS with 43.12% F.
    • Task 3 scores were very low. The best were ConcordU at 23.13% F for negation and 25.27% F for speculation.
  • Ensemble: voting over the top systems with U-Compare peaked at 55.96% F. This came from combining the top 6 systems with per-event-type weights.

Interesting ideas

  • I1. Split a complex task into layered subtasks. Making only the core layer (Task 1) mandatory lets participants enter at their own level. It also lets errors be attributed to specific layers, which would suit any structured extraction benchmark.
  • I2. Accept equivalent answers during scoring. The corpus marks alternative names for the same entity, and either name counts as correct. This avoids penalizing systems for legitimate variation and could help any task where several surface forms are valid.
  • I3. Fix argument typing before evaluating. The organizers required primary arguments to be proteins and moved details such as domains into secondary arguments like Site. This forces a consistent choice wherever the original annotation guidelines allowed several levels of specificity.
  • I4. Log a public online evaluator over time. Recording scores on development data throughout the challenge showed that systems were still improving at the deadline. That trajectory is useful for judging whether a benchmark is close to saturation.
  • I5. Run a post-hoc ensemble of submissions. Weighting each system's votes by its per-type performance revealed complementary strengths between systems. Development-set scores could supply those weights in practice, and the same check is cheap to run after any shared task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

147 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kim2009overview,
  title = {{Overview of BioNLP'09 Shared Task on Event Extraction}},
  author = {Kim, Jin-Dong and Ohta, Tomoko and Pyysalo, Sampo and Kano, Yoshinobu and Tsujii, Jun'ichi},
  booktitle = {Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task},
  year = {2009},
  doi = {10.3115/1572340.1572342},
  url = {https://doi.org/10.3115/1572340.1572342}
}

References

24 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gary D. Bader, Michael P. Cary, and Chris Sander. 2006. Pathguide: a Pathway Resource List. Nucleic Acids Research., 34(suppl 1):D504–506. find
  2. Daniel M. Bikel. 2004. Intricacies of Collins’ Parsing Model. Computational Linguistics, 30(4):479–511. find our card
  3. Evelyn Camon, Michele Magrane, Daniel Barrell, Vivian Lee, Emily Dimmer, John Maslen, David Binns, Nicola Harte, Rodrigo Lopez, and Rolf Apweiler. 2004. The Gene Ontology Annotation (GOA) Database: sharing knowledge in Uniprot with Gene Ontology. Nucl. Acids Res., 32(suppl 1):D262–266. find
  4. Eugene Charniak and Mark Johnson. 2005. Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173–180. find our card
  5. Andrew Chatr-aryamontri, Arnaud Ceol, Luisa Montecchi Palazzi, Giuliano Nardelli, Maria Victoria Schneider, Luisa Castagnoli, and Gianni Cesareni. 2007. MINT: the Molecular INTeraction database. Nucleic Acids Research, 35(suppl 1):D572–574. find
  6. Nancy Chinchor. 1998. Overview of MUC-7/MET-2. In Message Understanding Conference (MUC-7) Proceedings. find
  7. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), pages 449–454. find our card
  8. William Hersh, Aaron Cohen, Ruslenm Lynn, , and Phoebe Roberts. 2007. find
  9. TREC 2007 Genomics track overview. In Proceeding of the Sixteenth Text REtrieval Conference. find
  10. Lynette Hirschman, Martin Krallinger, and Alfonso Valencia, editors. 2007. Proceedings of the Second BioCreative Challenge Evaluation Workshop. CNIO Centro Nacional de Investigaciones Oncológicas. find
  11. Yoshinobu Kano, William Baumgartner, Luke McCrohon, Sophia Ananiadou, Kevin Cohen, Larry Hunter, and Jun’ichi Tsujii. 2009. U-Compare: share and compare text mining tools with UIMA. Bioinformatics. To appear. find
  12. Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA), pages 70–75. find our card
  13. Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from lterature. BMC Bioinformatics, 9(1):10. find
  14. David McClosky and Eugene Charniak. 2008. Self-Training for Biomedical Parsing. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics - Human Language Technologies (ACL-HLT’08), pages 101–104. find
  15. Claire Nédellec. 2005. Learning Language in Logic - Genic Interaction Extraction Challenge. In J. Cussens and C. Nédellec, editors, Proceedings of the 4th Learning Language in Logic Workshop (LLL05), pages 31– 37. find
  16. Tomoko Ohta, Jin-Dong Kim, Sampo Pyysalo, and Jun’ichi Tsujii. 2009. Incorporating GENETAG-style annotation to GENIA corpus. In Proceedings of Natural Language Processing in Biomedicine (BioNLP) NAACL 2009 Workshop. To appear. find
  17. Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Björne, Jorma Boberg, Jouni Järvinen, and Tapio Salakoski. 2007. BioInfer: A corpus for information extraction in the biomedical domain. BMC Bioinformatics, 8(50). find
  18. Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2009. Static Relations: a Piece in the Biomedical Information Extraction Puzzle. In Proceedings of Natural Language Processing in Biomedicine (BioNLP) NAACL 2009 Workshop. To appear. find
  19. Laura Rimell and Stephen Clark. 2008. Porting a lexicalized-grammar parser to the biomedical domain. Journal of Biomedical Informatics, To Appear. find
  20. Stephanie Strassel, Mark Przybocki, Kay Peterson, Zhiyi Song, and Kazuaki Maeda. 2008. Linguistic Resources and Evaluation Techniques for Evaluation of Cross-Document Automatic Content Extraction. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). find
  21. Lorraine Tanabe, Natalie Xie, Lynne Thom, Wayne Matten, and John Wilbur. 2005. Genetag: a tagged corpus for gene/protein named entity recognition. BMC Bioinformatics, 6(Suppl 1):S3. find
  22. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax Annotation for the GE-NIA corpus. In Proceedings of the IJCNLP 2005, Companion volume, pages 222–227. find
  23. Ellen Voorhees. 2007. Overview of TREC 2007. In The Sixteenth Text REtrieval Conference (TREC 2007) Proceedings. find
  24. John Wilbur, Lawrence Smith, and Lorraine Tanabe. 2007. BioCreative 2. Gene Mention Task. In L. Hirschman, M. Krallinger, and A. Valencia, editors, Proceedings of Second BioCreative Challenge Evaluation Workshop, pages 7–16. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2009-overview-bionlp-09-shared-task/