Elamur Research⌕
☰
home / papers / 2003 / Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Summary

The paper summarizes the CoNLL-2003 shared task on named entity recognition (NER), which was intended to be language-independent. The organizers prepared annotated corpora in English (news from the Reuters Corpus) and German (the newspaper Frankfurter Rundschau from the ECI Multilingual Text Corpus). Entities were annotated manually in four classes: PER, ORG, LOC and MISC. The data were additionally tokenized, part-of-speech tagged and chunked automatically. A large body of unannotated text was added to encourage participants to find ways of using it. Sixteen systems were evaluated with the F β=1 measure using exact matching of entity boundaries, and the significance of differences was tested with the bootstrap method. For both languages, the best performance came from the combined system of Florian et al., which merged several algorithms, gazetteers and two external NER recognizers. Majority voting among the top participants yielded an even better result. The CoNLL-2003 dataset subsequently became one of the main benchmarks for evaluating NER. For Ukrainian projects it is useful as a model of the IOB format, the class scheme and the evaluation protocol.

Key points

  • Data size: the English train set contains 14,987 sentences and 203,621 tokens, the German one 12,705 sentences and 206,931 tokens. The unannotated texts amount to about 17 million tokens in English and 14 million in German.
  • Format: each line contains one word and four columns (word, POS, chunk, NE tag) following the IOB scheme (Ramshaw & Marcus). Nested entities are generally not annotated.
  • Participants' methods: the most common was Maximum Entropy (5 systems). HMM, AdaBoost.MH, memory-based learning, SVM, CRF, LSTM and model ensembles were also used.
  • External resources: 11 of the 16 teams used additional data, and all of them used gazetteers. The unannotated texts gave a gain of only about 5%.
  • Organizers' ensemble: a vote across five systems reached F β=1 of 90.30 on English, cutting errors by 14% relative to the top single system, and 74.17 on German, a 6% error reduction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Named entity recognition

Cite

Cite as Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv cs/0306050

@inproceedings{sang2003introduction,
  title = {{Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition}},
  author = {Erik F. Tjong Kim Sang and Fien {De Meulder}},
  booktitle = {Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003},
  year = {2003},
  eprint = {cs/0306050},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/0306050}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Bender, Franz Josef Och, and Hermann Ney. 2003. Maximum Entropy Models for Named Entity Recognition. In Proceedings of CoNLL-2003. find
  2. Andrew Borthwick. 1999. A Maximum Entropy Approach to Named Entity Recognition. PhD thesis, New York University. find
  3. Xavier Carreras, Lluı́s Màrquez, and Lluı́s Padró. 2003a. Learning a Perceptron-Based Named Entity Chunker via Online Recognition Feedback. In Proceedings of CoNLL-2003. find
  4. Xavier Carreras, Lluı́s Màrquez, and Lluı́s Padró. 2003b. A Simple Named Entity Extractor using AdaBoost. In Proceedings of CoNLL-2003. find
  5. Rich Caruana and Dayne Freitag. 1994. Greedy Attribute Selection. In Proceedings of the Eleventh International Conference on Machine Learning, pages 28–36. New Brunswick, NJ, USA, Morgan Kaufman. find
  6. Hai Leong Chieu and Hwee Tou Ng. 2003. Named Entity Recognition with a Maximum Entropy Approach. In Proceedings of CoNLL-2003. find
  7. Nancy Chinchor, Erica Brown, Lisa Ferro, and Patty Robinson. 1999. 1999 Named Entity Recognition Task Definition. MITRE and SAIC. find
  8. James R. Curran and Stephen Clark. 2003. Language Independent NER using a Maximum Entropy Tagger. In Proceedings of CoNLL-2003. find
  9. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2002. MBT: Memory-Based Tagger, version 1.0, Reference Guide. ILK Technical Report ILK-0209, University of Tilburg, The Netherlands. find
  10. Fien De Meulder and Walter Daelemans. 2003. Memory-Based Named Entity Recognition using Unannotated Data. In Proceedings of CoNLL-2003. find
  11. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named Entity Recognition through Classifier Combination. In Proceedings of CoNLL-2003. our card
  12. James Hammerton. 2003. Named Entity Recognition with Long Short-Term Memory. In Proceedings of CoNLL-2003. find
  13. Iris Hendrickx and Antal van den Bosch. 2003. Memory-based one-step named-entity recognition: Effects of seed list features, classifier stacking, and unannotated data. In Proceedings of CoNLL-2003. find
  14. Dan Klein, Joseph Smarr, Huy Nguyen, and Christopher D. Manning. 2003. Named Entity Recognition with Character-Level Models. In Proceedings of CoNLL-2003. find
  15. James Mayfield, Paul McNamee, and Christine Piatko. 2003. Named Entity Recognition using Hundreds of Thousands of Features. In Proceedings of CoNLL-2003. find
  16. Andrew McCallum and Wei Li. 2003. Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons. In Proceedings of CoNLL-2003. our card
  17. Robert Munro, Daren Ler, and Jon Patrick. 2003. Meta-Learning Orthographic and Contextual Models for Language Independent Named Entity Recognition. In Proceedings of CoNLL-2003. find
  18. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses. John Wiley & Sons. find
  19. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text Chunking Using Transformation-Based Learning. In Proceedings of the Third ACL Workshop on Very Large Corpora, pages 82–94. Cambridge, MA, USA. our card
  20. Helmut Schmid. 1995. Improvements in Part-of-Speech Tagging with an Application to German. In Proceedings of EACL-SIGDAT 1995. Dublin, Ireland. find
  21. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of CoNLL-2002, pages 155–158. Taipei, Taiwan. our card
  22. C.J. van Rijsbergen. 1975. Information Retrieval. Buttersworth. find
  23. English test Florian Chieu Klein Zhang Carreras (a) Curran Mayfield Carreras (b) McCallum Bender Munro Wu Whitelaw Hendrickx De Meulder Hammerton Baseline Precision 88.99% 88.12% 85.93% 86.13% 84.05% 84.29% 84.45% 85.81% 84.52% 84.68% 80.87% 82.02% 81.60% 76.33% 75.84% 69.09% 71.91% Recall 88.54% 88… find
  24. Casey Whitelaw and Jon Patrick. 2003. Named Entity Recognition Using a Character-based Probabilistic Approach. In Proceedings of CoNLL-2003. find
  25. Dekai Wu, Grace Ngai, and Marine Carpuat. 2003. A Stacked, Voted, Stacked Model for Named Entity Recognition. In Proceedings of CoNLL-2003. find
  26. Tong Zhang and David Johnson. 2003. A Robust Risk Minimization based Named Entity Recognition System. In Proceedings of CoNLL-2003. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/