Elamur Research⌕
☰
home / papers / 2011 / Overview of BioNLP Shared Task 2011

Overview of BioNLP Shared Task 2011

Paper · 2011 · Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta et al.

Summary

BioNLP-ST 2011 was the second round of a community challenge on fine-grained biomedical information extraction, especially event extraction. The 2009 edition relied only on the Genia corpus, which covers one narrow subdomain and comes from one group. This edition was run jointly by several groups to test whether extraction methods generalize in three directions: text type, event type, and subject domain. It offered five main tasks: GE, EPI, ID, and the bacteria track tasks BB and BI. It also offered three supporting tasks meant to help with them: CO (protein coreference), REL (entity relations) and REN (gene renaming). Over about six months, 24 teams sent 46 final submissions. On the repeated GE abstract task, systems cut errors by more than 10% compared with 2009. Performance on full papers and new domains fell only slightly. BI beat the earlier LLL result by about 20 F-score points. One goal failed: no team used supporting-task outputs in its main-task systems.

Key points

  • Text types widened from abstracts to full papers (GE, ID) and web pages (BB). The number of targeted event/relation types was GE 9, EPI 15, ID 10, BB 2 and BI 10.
  • EPI added 14 event types (protein/DNA modifications and their reversals) and two new argument roles, Sidechain and Contextgene. ID added four entity types, including CHEMICAL and ORGANISM, plus a category for high-level biological processes.
  • The best GE abstract F-score rose to 57.46, from 51.95 in 2009. Full papers reached 53.14, and three teams beat the strongest earlier result (Miwa et al., 2010b).
  • Per the results table, other best F-scores were roughly IDf 55.59, EPIf 53.33, BB 45, BI 77, CO 34.05, REL 57.70 and REN 87.00.
  • UTurku was the only team to enter all eight tasks. Seventeen of the 24 teams entered just one task.

Interesting ideas

  • I1. Keep a frozen benchmark next to new data. GE reused the exact 2009 task definition and abstract set while adding a full-paper set. This lets one task measure both year-over-year progress and transfer to a new text type, which is useful for any recurring benchmark, such as a Ukrainian NLP evaluation.
  • I2. Use subtask views to isolate difficulty. Comparing the same event type across tasks (GE vs EPI PHOSPHORYLATION) shows whether a domain shift itself hurts results. Splitting tasks into core and full versions separates trigger and core-argument extraction from extra-argument extraction.
  • I3. Choose supporting tasks from error analysis. CO was created because the 2009 analysis showed coreference was blocking event extraction. Building auxiliary tasks around measured failure points is a principled way to design a challenge.
  • I4. Stage the schedule. Supporting tasks ran and finished first so their outputs could feed the main tasks. The paper reports this produced no actual uptake and suggests the long, complex setup overloaded teams. The lesson is that organizers should keep the burden low if they want participants to combine resources.
  • I5. Provide gold entities to narrow the focus. Giving gold named-entity annotation lets participants concentrate on relation and event extraction. BB, which required teams to recognize entities themselves, scored lower, which shows the cost of removing that help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft Research AsiaBeijing · China54 works · 59,921 citations🎓The University of TokyoTokyo · Japan45 works · 8,692 citations

Also on the byline: Database Center for Life Science.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Biochemistry, Genetics and Molecular Biology · 116 worksMolecular Biology · 114 worksDecision Sciences · 85 worksInformation Systems and Management · 18 works

Cite

Cite as Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta, Robert Bossy, Ngan Nguyen and Jun'ichi Tsujii. 2011. Overview of BioNLP Shared Task 2011. Proceedings of BioNLP Shared Task 2011 Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kim2011overview,
  title = {{Overview of BioNLP Shared Task 2011}},
  author = {Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta, Robert Bossy, Ngan Nguyen and Jun'ichi Tsujii},
  booktitle = {Proceedings of BioNLP Shared Task 2011 Workshop},
  year = {2011},
  url = {https://aclanthology.org/W11-1801/}
}

References

34 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sophia Ananiadou, Sampo Pyysalo, Junichi Tsujii, and Douglas B. Kell. 2010. Event extraction for systems biology by text mining the literature. Trends in Biotechnology. find
  2. Sophia Ananiadou, Dan Sullivan, William Black, Gina-Anne Levow, Joseph J. Gillespie, Chunhong Mao, Sampo Pyysalo, BalaKrishna Kolluru, Junichi Tsujii, and Bruno Sobral. 2011. Named entity recognition for bacterial type IV secretion systems. PLoS ONE, 6(3):e14780. find
  3. Jari Björne, Filip Ginter, Sampo Pyysalo, Jun’ichi Tsujii, and Tapio Salakoski. 2010. Complex event extraction at PubMed scale. Bioinformatics, 26(12):i382–390. find
  4. Robert Bossy, Julien Jourde, Philippe Bessières, Marteen van de Guchte, and Claire Nédellec. 2011. find
  5. BioNLP Shared Task 2011 - Bacteria Biotope. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  6. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó. Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th Internationa…
  7. Lynette Hirschman, Martin Krallinger, and Alfonso Valencia, editors. 2007. Proceedings of the Second BioCreative Challenge Evaluation Workshop. CNIO Centro Nacional de Investigaciones Oncológicas. find
  8. Julien Jourde, Alain-Pierre Manine, Philippe Veber, Karën Fort, Robert Bossy, Erick Alphonse, and Philippe Bessières. 2011. find
  9. BioNLP Shared Task 2011 - Bacteria Gene Interactions and Renaming. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  10. Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from lterature. BMC Bioinformatics, 9(1):10. find
  11. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on Event Extraction. In Proceedings of Natural Language Processing in Biomedicine (BioNLP) NAACL 2009 Workshop, pages 1–9.
  12. Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of the Genia Event task in BioNLP Shared Task 2011. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  13. A.P. Manine, E. Alphonse, and Bessières P. 2009. Learning ontological rules to extract multiple relations of genic interactions from text. International Journal of Medical Informatics, 78(12):e31–38. find
  14. Makoto Miwa, Sampo Pyysalo, Tadayoshi Hara, and Jun’ichi Tsujii. 2010a. A comparative study of syntactic parsers for event extraction. In Proceedings of BioNLP’10, pages 37–45. find
  15. Makoto Miwa, Rune Sætre, Jin-Dong Kim, and Jun’ichi Tsujii. 2010b. Event extraction with complex event classification using rich features. Journal of Bioinformatics and Computational Biology (JBCB), 8(1):131– 146, February. find
  16. Nédellec. 2005. Learning Language in Logic – Genic Interaction Extraction Challenge. In Proceedings of 4th Learning Language in Logic Workshop (LLL’05), pages 31–37. find
  17. Ngan Nguyen, Jin-Dong Kim, and Jun’ichi Tsujii. 2011. Overview of the Protein Coreference task in BioNLP Shared Task 2011. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  18. Tomoko Ohta, Sampo Pyysalo, Jin-Dong Kim, and Jun’ichi Tsujii. 2010a. A re-evaluation of biomedical named entity-term relations. Journal of Bioinformatics and Computational Biology (JBCB), 8(5):917–928. find
  19. Tomoko Ohta, Sampo Pyysalo, Makoto Miwa, Jin-Dong Kim, and Jun’ichi Tsujii. 2010b. Event extraction for post-translational modifications. In Proceedings of BioNLP’10, pages 19–27. find
  20. Tomoko Ohta, Sampo Pyysalo, Makoto Miwa, and Jun’ichi Tsujii. 2010c. Event extraction for dna methylation. In Proceedings of SMBM’10. find
  21. Tomoko Ohta, Sampo Pyysalo, and Jun’ichi Tsujii. 2011. Overview of the Epigenetics and Post-translational Modifications (EPI) task of BioNLP Shared Task 2011. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistic… find
  22. Maté Ongenaert, Leander Van Neste, Tim De Meyer, Gerben Menschaert, Sofie Bekaert, and Wim Van Criekinge. 2008. PubMeth: a cancer methylation database combining text-mining and expert annotation. Nucleic Acids Research, 36(suppl 1):D842–846. find
  23. Hoifung Poon and Lucy Vanderwende. 2010. Joint inference for knowledge extraction from biomedical literature. In Proceedings of NAACL-HLT’10, pages 813– 821. find
  24. Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2009. Static Relations: a Piece in the Biomedical Information Extraction Puzzle. In Proceedings of Natural Language Processing in Biomedicine (BioNLP) NAACL 2009 Workshop, pages 1–9, Boulder, Colorado. Association for Computational Lingu… find
  25. Sampo Pyysalo, Tomoko Ohta, Han-Cheol Cho, Dan Sullivan, Chunhong Mao, Bruno Sobral, Jun’ichi Tsujii, and Sophia Ananiadou. 2010. Towards event extraction from full texts on infectious diseases. In Proceedings of BioNLP’10, pages 132–140. find
  26. Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Dan Sullivan, Chunhong Mao, Chunxia Wang, Bruno Sobral, Jun’ichi Tsujii, and Sophia Ananiadou. 2011a. Overview of the Infectious Diseases (ID) task of BioNLP Shared Task 2011. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland… find
  27. Sampo Pyysalo, Tomoko Ohta, and Jun’ichi Tsujii. 2011b. Overview of the Entity Relations (REL) supporting task of BioNLP Shared Task 2011. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  28. Pontus Stenetorp, Goran Topić, Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2011. find
  29. BioNLP Shared Task 2011: Supporting Resources. In Proceedings of the BioNLP 2011 Workshop Companion Volume for Shared Task, Portland, Oregon, June. Association for Computational Linguistics. find
  30. Jian Su, Xiaofeng Yang, Huaqing Hong, Yuka Tateisi, and Jun’ichi Tsujii. 2008. Coreference Resolution in Biomedical Texts: a Machine Learning Approach. In Ontologies and Text Mining for Life Sciences’08. find
  31. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax Annotation for the GE-NIA corpus. In Proceedings of the IJCNLP 2005, Companion volume, pages 222–227. find
  32. Andreas Vlachos. 2010. Two strong baselines for the bionlp 2009 event extraction task. In Proceedings of BioNLP’10, pages 1–9. find
  33. Cathy H. Wu, Lai-Su L. Yeh, Hongzhan Huang, Leslie Arminski, Jorge Castro-Alvear, Yongxing Chen, Zhangzhi Hu, Panagiotis Kourtesis, Robert S. Ledley, Baris E. Suzek, C.R. Vinayaka, Jian Zhang, and Winona C. Barker. 2003. The Protein Information Resource. Nucleic Acids Research, 31(1):345–347. find
  34. H. Yu and E. Agichtein. 2003. Extracting synonymous gene and protein terms from biological literature. Bioinformatics, 19(suppl 1):i340. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2011-overview-bionlp-shared-task-2011/