Elamur Research⌕
☰
home / papers / 2011 / Overview of BioNLP 2011 Protein Coreference Shared Task

Overview of BioNLP 2011 Protein Coreference Shared Task

Summary

This overview describes the Protein Coreference (COREF) task, a supporting task in BioNLP Shared Task 2011. Earlier shared-task experience showed that anaphora in biomedical abstracts blocks fine-grained event extraction, so COREF asked systems to link pronouns, relative pronouns and definite noun phrases to the protein names they refer to. The organizers built benchmark data from the Genia corpus by combining MedCO coreference annotation, Genia event annotation (for gold protein names) and the Genia Treebank (for base NP boundaries), keeping only anaphors that could plausibly denote proteins. Systems were scored in two modes. A surface mode checked links as annotated. The primary protein coreference mode followed link chains until it reached an antecedent containing a protein name. Six teams submitted results after a seven-week development period. The best system recovered 22.18% of protein coreference links at 73.26% precision. The error analysis shows that relative pronouns are handled reasonably well, while definite NPs, pronouns, indirect links and anaphoricity detection remain largely unsolved. These findings point to clear priorities for biomedical coreference research.

Key points

  • Data sources and size: the data sets come from MedCO, Genia event annotation and Genia Treebank, all layered on the Genia corpus.
    • The training set contains 4,363 coreference entities, including 1,193 relative pronouns, 738 pronouns, 296 definite/demonstrative NPs and 2,116 antecedents.
    • The development set contains 914 entities and the test set contains 1,388.
  • Two evaluation modes: systems were scored with precision, recall and F-score under partial boundary matching (minimal/maximal spans, following MUC).
    • The surface mode scored the annotated links directly.
    • The protein coreference mode was primary. It counted both direct and indirect (chained) links to actual protein mentions.
  • Participation and top result: six teams took part (UofU, UZH, ConcordU, UTurku, USzeged, UCD SCI), using both rule-based and machine learning approaches. The top system reached 22.18% recall at 73.26% precision in the primary mode.
  • Performance by anaphor type: relative pronouns were the easiest category, with exact recall up to 77.5%.
    • For definite NPs, best recall stayed below 6%, even though they account for nearly a fifth of protein links. Only one team (UCD SCI) attempted them.
    • Pronoun resolution also stayed weak.
  • Main bottlenecks: few indirect links were found, and anaphoricity determination was poor (best F about 34% against 210 gold anaphors). The authors identify these as the main obstacles to better results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nguyen2011overview,
  title = {{Overview of BioNLP 2011 Protein Coreference Shared Task}},
  author = {Nguyen, Ngan and Kim, Jin-Dong and Tsujii, Jun'ichi},
  booktitle = {Proceedings of BioNLP Shared Task 2011 Workshop},
  year = {2011},
  url = {https://aclanthology.org/W11-1811/}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Baldwin. 1997. Cogniac: High precision with limited knowledge and linguistic resources. In Proceedings of the ACL’97/EACL’97 Workshop on Operational Factors in Practical, Robust Anaphora Resolution, pages 38–45, Madrid, Spain. find
  2. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala, Sweden, July. Association for Computational Linguistics. find our card
  3. Nancy Chinchor. 1998. Overview of MUC-7/MET-2. In Message Understanding Conference (MUC-7) Proceedings. find
  4. Yu hsiang Lin and Tyne Liang. 2004. Pronominal and sortal anaphora resolution for biomedical literature. In In Proceedings of ROCLING XVI: Conference on Computational Linguistics and Speech Processing. find our card
  5. Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from lterature. BMC Bioinformatics, 9(1):10. find
  6. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on Event Extraction. In Proceedings of Natural Language Processing in Biomedicine (BioNLP) NAACL 2009 Workshop, pages 1–9. find
  7. Makoto Miwa, Rune Sætre, Jin-Dong Kim, and Jun’ichi Tsujii. 2010. Event extraction with complex event classification using rich features. Journal of Bioinformatics and Computational Biology (JBCB), 8(1):131– 146, February. find
  8. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of the ACL, pages 1396–1411. find our card
  9. Ngan Nguyen and Jin-Dong Kim. 2008. Exploring domain differences for the design of a pronoun resolution system for biomedical texts. In Proceedings of 22nd International Conference on Computational Linguistics (COLING-2008). find
  10. T Ohta, Y Tateisi, H Mima, and J Tsujii. 2002. Genia corpus: an annotated research abstract corpus in molecular biology domain. Proceedings of the Human Language Technology Conference (HLT 2002), San Diego, California, pages 73–77. find
  11. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nate Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 492–501, October. find our card
  12. V. Stoyanov, C. Cardie, N. Gilbert, E. Riloff, D. Buttler, and D. Hysom. 2010a. Coreference resolution with reconcile. In Proceedings of the Conference of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010). find our card
  13. V. Stoyanov, C. Cardie, N. Gilbert, E. Riloff, D. Buttler, and D. Hysom. 2010b. Reconcile: A coreference resolution platform. In Tech Report - Cornell University. find
  14. Stephanie Strassel, Mark Przybocki, Kay Peterson, Zhiyi Song, and Kazuaki Maeda. 2008. Linguistic Resources and Evaluation Techniques for Evaluation of Cross-Document Automatic Content Extraction. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). find
  15. Jian Su, Xiaofeng Yang, Huaqing Hong, Yuka Tateisi, and Jun’ichi Tsujii. 2008. Coreference Resolution in Biomedical Texts: a Machine Learning Approach. In Ontologies and Text Mining for Life Sciences’08. find
  16. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax annotation for the genia corpus. In International Joint Conference on Natural Language Processing, pages 222–227, Jeju Island, Korea, October. find

↑ back to top

This page in Markdown →