Elamur Research
☰
home / papers / 2004 / Introduction to the Bio-entity Recognition Task at JNLPBA

Introduction to the Bio-entity Recognition Task at JNLPBA

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W04-1213.pdf

Summary

This paper introduces the JNLPBA 2004 shared task on bio-entity recognition: automatically locating and labelling names of molecular-biology concepts in MEDLINE abstracts. Such systems had trailed newswire NER by roughly 30 F-score points. Reasons include unclear entity boundaries, abbreviations and aliases, and the difficulty of annotating many classes consistently. The organisers built a common benchmark from GENIA version 3.02, reducing its 36 terminal classes to five types and providing the data in IOB2 format. For testing they used a fresh, biologist-checked set of abstracts, split by publication period and by domain so that generalisation could be measured. Eight teams took part, with no restrictions on methods or resources. Scoring used a modified CoNLL script. The strongest systems paired robust learners (SVM, MEMM, CRF) with rich features, abbreviation handling and external resources such as gazetteers. Even so, the best total F-score stayed below 73. The task became a widely used reference benchmark for biomedical NER.

Key points

  • Training data: 2,000 GENIA 3.02 abstracts, gathered through a MeSH-based search combining 'transcription factors', 'blood cells' and 'human'. They contain 472,006 words and 51,301 entities in five classes: protein, DNA, RNA, cell type and cell line. Nested annotations were flattened to the outermost span.
  • Test data: 404 abstracts with 8,662 entities. They were split into 1978-1989, 1990-1999, 2000-2001 and super-domain (S/1998-2001) subsets, the last retrieved without the term 'human'.
  • Evaluation: F-score is reported under exact matching and also under left-only and right-only boundary matching.
  • Results: SVMs were used by five of the eight systems, sometimes combined with HMMs or CRFs. The top total score was 72.6 F (76.0 recall / 69.4 precision). Most systems did best on the 1990-1999 subset, the one closest to the training data.
  • Notable observations: Settles' CRF system came close to the leaders while using a fairly simple feature set. Web-based resources were starting to appear as a source of features. Note that the paper lists Jin-Dong Kim as first author, with Nigel Collier as a co-author.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Life sciences & medicine

Topics: Named entity recognition

Cited by

158 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 153 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{collier2004introduction,
  title = {{Introduction to the Bio-entity Recognition Task at JNLPBA}},
  author = {Collier, Nigel and Ohta, Tomoko and Tsuruoka, Yoshimasa and Tateisi, Yuka and Kim, Jin-Dong},
  booktitle = {Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP)},
  year = {2004},
  url = {https://aclanthology.org/W04-1213/}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jin-Dong Kim, Tomoko Ohta, Yuka Tateishi, and Jun’ichi Tsujii. 2003. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19(Suppl.1):180–182. find
  2. DARPA. 1995. Proceedings of the Sixth Message Understanding Conference(MUC-6), Columbia, MD, USA, November. Morgan Kaufmann. find
  3. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning (CoNLL-2003), pages 142–147. Edmonton, Canada. find
  4. C. J. van Rijsbergen. 1979. Information Retrieval. Butterworths, London. find
  5. Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi and Jun’ichi Tsujii. 2002. Corpus-Based Approach to Biological Entity Recognition. in Proceedings of the Second Meeting of the Special Interest Group on Test Data Mining of ISMB (BioLink-2002), Edmonton, Canada. find
  6. GuoDong Zhou and Jian Su. 2004. Exploring Deep Knowledge Resources in Biomedical Name Recognition. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  7. Jenny Finkel, Shipra Dingare, Huy Nguyen, Malvina Nissim, Gail Sinclair and Christopher Manning. 2004. Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004),… find
  8. Burr Settles. 2004. Biomedical Named Entity Recognition Using Conditional Random Fields and Novel Feature Sets. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  9. Yu Song, Eunju Kim, Gary Geunbae Lee and Byoung-kee Yi. 2004. POSBIOTM-NER in the shared task of BioNLP/NLPBA 2004. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  10. Shaojun Zhao. 2004. Name Entity Recognition in Biomedical Text using a HMM model in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  11. Marc Rössler. 2004. Adapting an NER-System for German to the Biomedical Domain. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  12. Kyung-Mi Park, Seon-Ho Kim, Do-Gil Lee and Hae-Chang Rim. 2004. Boosting Lexical Knowledge for Biomedical Named Entity Recognition. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find
  13. Chih Lee, Wen-Juan Hou and Hsin-Hsi Chen. 2004. Annotating Multiple Types of Biomedical Entities: A Single Word Classificica-tion Approach. in Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA-2004), Geneva, Switzerland. find