Elamur Research⌕
☰
home / papers / 2000 / Introduction to the CoNLL-2000 Shared Task Chunking

Introduction to the CoNLL-2000 Shared Task Chunking

Summary

The paper presents the CoNLL-2000 shared task on text chunking, which means splitting sentences into non-overlapping, syntactically coherent word groups such as noun, verb and prepositional phrases. Earlier work concentrated almost entirely on base noun phrases. This task extends the problem to eleven chunk types. No corpus was annotated directly for arbitrary chunks, so the organizers derived chunks automatically from parse trees in the WSJ portion of the Penn Treebank II. The rule they used is that a chunk runs from its left edge up to and including the syntactic head, with premodifiers but no arguments or postmodifiers. Sections 15–18 served for training and section 20 for testing. Tokens carry POS tags from the Brill tagger rather than gold tags, and chunks are encoded with B/I/O labels. Eleven participating systems were compared: rule-based, memory-based, statistical and combined. All of them beat a simple POS-based baseline. The top system, an SVM combination by Kudoh and Matsumoto, reached F(β=1) = 93.48. The resulting dataset and conversion script became a standard benchmark for shallow parsing.

Key points

  • Data: WSJ sections 15–18 of the Penn Treebank II for training and section 20 for testing, the same split as the Ramshaw and Marcus (1995) base-NP data. The training set has 211,727 tokens and 106,978 chunks.
  • Chunk types (11): NP (51% of chunks), VP (20%), PP (20%), ADVP (4%), SBAR, ADJP, PRT, CONJP, INTJ, LST and UCP. UCP is the rarest, with only 2 instances in training.
  • Encoding: each token gets a B-X tag (chunk-initial), an I-X tag (inside a chunk) or O (outside any chunk). This turns chunking into a per-word tagging problem.
  • Evaluation: precision, recall and F(β=1), with the F-score as the optimization target. The baseline assigns each POS tag its most frequent chunk tag.
  • Systems: eleven in four families: rule-based, memory-based, statistical (Markov models/HMMs, maximum entropy) and combined. The winner, Kudoh & Matsumoto, used 231 SVM classifiers merged by dynamic programming and scored F = 93.48.

Interesting ideas

  • I1. Deriving a new task from an existing treebank: the chunk annotation was produced by a script from existing parse trees, with a head-based rule deciding chunk boundaries. Any language with a treebank could build a cheap shallow-parsing dataset the same way without fresh manual annotation.
  • I2. Releasing the conversion script: the configurable converter was published with the data, so others can change design decisions and measure how those choices affect task difficulty. This makes annotation conventions an object of study rather than a fixed assumption.
  • I3. Predicted POS tags instead of gold tags: inputs were tagged by a standard POS tagger instead of using treebank tags. Scores therefore reflect realistic conditions on raw text, which is a sensible default for any pipeline benchmark.
  • I4. IOB encoding of spans: representing non-overlapping spans as per-token B/I/O labels lets any sequence tagger handle segmentation. The same trick applies to NER, multiword expressions or address parsing.
  • I5. Ensembles over different output representations: several top systems gained by combining classifiers, for example by majority voting over learners trained on different chunk-tag encodings. Combining diverse views of the same task is a cheap way to beat any single model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Treebanks · Part-of-speech tagging

Cited by

276 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 259 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sang2000introduction,
  title = {{Introduction to the CoNLL-2000 Shared Task Chunking}},
  author = {Sang, Erik F. Tjong Kim and Buchholz, Sabine},
  booktitle = {Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop},
  year = {2000},
  url = {https://aclanthology.org/W00-0726/}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney. 1991. Parsing by chunks. In Principle-Based Parsing. Kluwer Academic Publishers. find
  2. Ann Bies, Mark Ferguson, Karen Katz, and Robert MacIntyre. 1995. Bracket Guidelines /or Treebank H Style Penn Treebank Project. Penn Treebank II cdrom. find
  3. Eric Brill. 1994. Some advances in rule-based part of speech tagging. In Proceedings o] the Twelfth National Con/erence on Artificial Intelligence (AAAI-9~). Seattle, Washington. find
  4. Sabine Buchholz, Jorn Veenstra, and Walter Daelemans. 1999. Cascaded grammatical :relation assignment. In Proceedings o] EMNLP/VLC-99. Association for Computational Linguistics. find
  5. John Carroll, Ted Briscoe, Glenn Carroll, Marc Light, Dethleff Prescher, Mats Rooth, Stefano Federici, Simonetta Montemagni, Vito Pirrelli, Irina Prodanof, and Massimo Vanocchi. 1997. Phrasal Parsing Software. Sparkle Work Package 3, Deliverable D3.2. find
  6. Kenneth Ward Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Con]erence on Applied Natural Language Processing. Austin, Texas. find
  7. H@rve D@jean. 2000. Learning syntactic structures with xml. In Proceedings o/ CoN~LL-2000 and LLL-2000. Lisbon, Portugal. find
  8. Christer Johansson. 2000. A context sensitive maximum likelihood approach to chunking. In Proceedings o] CoNLL-2000 and LLL-2000. find
  9. Lisbon, Portugal. Rob Koeling. 2000. Chunking with maximum entropy models. In Proceedings o/ CoNLL-2000 and LLL-2000. find
  10. Lisbon, Portugal. Taku Kudoh and Yuji Matsumoto. 2000. Use of support vector learning for chunk identification. In Proceedings o~ CoNLL-2000 and LLL-2000. find
  11. Lisbon, Portugal. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2). find our card
  12. Miles Osborne. 2000. Shallow parsing as part-of-speech tagging. In Proceedings o] CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  13. Ferran Pla, Antonio Molina, and Natividad Prieto. 2000. Improving chunking by means of lexical-contextual information in statistical language models. In Proceedings o] CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  14. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings o] the Third A CL Workshop on Very Large Corpora. Association for Computational Linguistics. find
  15. Adwait Ratnaparkhi. 1998. Maximum Entropy Models ]or Natural Language Ambiguity Resolution. PhD thesis Computer and Information Science, University of Pennsylvania. find
  16. Erik F. Tjong Kim Sang. 2000. Text chunking by system combination. In Proceedings of CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  17. Hans van Halteren. 2000. Chunking with wpdv models. In Proceedings o/ CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  18. C.J. van Rijsbergen. 1975. In/ormation Retrieval. Buttersworth. find
  19. Jorn Veenstra and Antal van den Bosch. 2000. Single-classifier memory-based phrase chunking. In Proceedings o] CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  20. Jorn Veenstra. 1999. Memory-based text chunking. In Nikos Fakotakis, editor, Machine learning in human language technology, workshop at ACAI 99. find
  21. Marc Vilain and David Day. 2000. Phrase parsing with rule sequence processors: an application to the shared conll task. In Proceedings o/CoNLL-2000 and LLL-2000. Lisbon, Portugal. find
  22. GuoDong Zhou, Jian Su, and TongGuan Tey. 2000. Hybrid text chunking. In Proceedings o/CoNLL-2000 and LLL-2000. Lisbon, Portugal. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/sang-2000-introduction-conll-2000-shared-task/